Emergent World Models and Latent Variable Estimation in Chess-Playing Language Models

Thank you for your thorough review and insightful comments.

Incremental nature

Our research contributes to the debate surrounding the robust world models found in OthelloGPT. OthelloGPT only had a robust world model when trained on a large synthetic dataset and did not find a robust world model when trained on human games, and there was debate on if this was due to the size or synthetic nature of the dataset.

We emphasize the significance of using chess as a testbed for studying steering vector mechanisms, as chess provides measurable ground truth unlike natural language. Mechanistic interpretability is severely hampered by the lack of a ground truth in language models. We aim to further this promising direction.

Question on linear probes.

We agree that clarification is needed. We propose modifying the paragraph as follows: "To use this objective, for every layer, we must train one group of 64 linear probes for predicting the board state at every white move and a second group of 64 linear probes at every black move, rather than a single group of 64 linear probes at every move."

F1 score for probe outputs We agree that measuring F1 score on a per-class basis would be a valuable addition to the paper. Although we cannot submit revisions during the discussion process, we have ran the experiments and will incorporate this analysis in the final version of the paper.

Gradient descent We appreciate your thoughts on the gradient descent approach used by Li et al. (2023). In our initial experiments, we found that linear probe addition outperformed gradient descent by several percent, similar to Nanda et al's results.

Skill vector effects Your interpretation of the model imitating a player according to the estimated skill level is spot-on. In the case of a game initialized with random moves, a good next-token predictor would predict legal random moves, which is what we find. By adding the skill vector, we can fool the model to imitate a stronger player, even if the game begins with random moves. We appreciate the connection to Andreas.

Board state interventions, Table 2, no intevention - Modified Board

Your understanding is correct. No intervention - modified board serves as a baseline in this case. We are evaluating the original model on the make-believe modified board, which has one piece deleted from it. In this case, 60% of the original model’s moves are illegal because it is attempting to move a deleted piece.

Thank you for your positive feedback and insightful questions. We appreciate your recognition of the paper's quality, clarity, and significant contribution to understanding what language models learn beyond surface-level statistics.

Elo probe range

We chose the range of 25-35 turns as it was far enough along in the game to have sufficient information from which player skill could be inferred. If we use all turns (such as turns 0-35), the linear probe's accuracy decreases. This is because there is little information in early game moves that can reliably predict player skill. For example, it is nearly impossible to predict player skill after the first turn. If we use fewer turns or a constant number, such as only using data from turn 25, we would need approximately 10 times as many games for the linear probe to obtain the same amount of training data as using the 25-35 turn range.

Layer performance difference

One hypothesis is that the model computes a board state and probability distribution over next moves, and then uses its skill estimate to shift the probability distribution. In appendix C, skill interventions are typically most successful in later layers compared to board state interventions, offering tentative evidence towards this hypothesis. A more careful examination of intervention success by intervention type and layer number would be helpful. It would be best to use mechanistic interpretability techniques to further understand how the skill estimation and board state interact, but we leave this to future work.

Figure 3, jump at layer 7

Our hypothesis is that the model has to process the full sequence of moves and game history in order to compute the board state. It may be able to first estimate player Elo while processing the game history, before board state is fully computed. We notice that in the 8-layer model result, it has a jump in elo classification at layer 5, before peak board state accuracy is achieved at layer 6.

Thank you again for your valuable feedback.

Thank you for your insightful review and positive assessment of our paper. We appreciate your feedback and the opportunity to clarify a few points.

Implications for other models

Crafting steering vectors as demonstrated in the paper has broad implications for the NLP community. Steering vectors are interesting and have powerful effects on LLM behavior, such as increasing truthfulness or reducing hallucinations, but it is difficult to mechanistically study how they are modifying the LLM’s behavior as there is no underlying measurable ground truth in natural language. This is why we believe chess could serve as a valuable testbed for studying the mechanistic interpretability of steering vectors in LLMs and for mechanistic interpretability in general. In chess, there is an underlying ground truth we can measure for the world state, and Stockfish can be used as an oracle to grade the quality of moves after an intervention. The existence of this ground truth makes it much more tractable to understand what is going on internally.

Clarification of board state intervention

Intuitively, for the board state intervention, we edit the model’s internal board state (we do not alter the input PGN string) and if our intervention is successful, the model’s move will be legal under the new make-believe board state.

We begin with the original model’s deterministic next move based on a fixed PGN string. We identify the piece P the model intends to move during this first forward pass. Next, we perform another forward pass through the model, but this time we subtract the steering vector for P from the model’s activations. Conceptually, this deletes the piece from the model’s internal world representation. We sample from this intervened forward pass for 5 subsequent moves. If successful, the five moves should all be legal despite our intervention creating a make-believe state within the model’s internal world representation. Finally, we check that the moves are all legal using a separate chess library which we initialized from the FEN representation of the make-believe board where P was removed.

When we delete the piece P above, we check that the modified board is still valid and has at least one legal move available before sampling from the model.

Thank you for pointing out the minor wording issues. We will clarify the linear representation sentence and revise the 2017 research sentence.

Thank you for the thorough review and feedback. We have addressed your key points below.

Significance of Results

Our research contributes to the debate on robust world models in OthelloGPT, which only emerged with large synthetic datasets. There were questions if this was due to size or synthetic nature of the data.

We emphasize the significance of the ELO intervention, demonstrating chess as a testbed for studying steering vector mechanisms, which are also applicable to latent variables like personality and style in LLMs. Chess provides measurable ground truth unlike natural language, making study of these mechanisms tractable.

While OthelloGPT and ChessGPT can track board state, the how is not understood. ChessGPT enables deeper study with greater strategic complexity while maintaining an available ground truth. In general, interpretability is greatly hampered by lack of an existing ground truth in natural language.

Elo Probe Range The Elo probe works outside its training range. In 4.3, it accurately classifies games with 20 random initial moves as low skill on turn 10 (below the 25-35 training range).

Board State Intervention

We edit the model's internal state (not PGN input). Thus the input PGN is invalid relative to the desired board state. If successful, the model's move is legal under the new "make-believe" state.

We subtract the steering vector for piece P to "delete" it in the model's internal representation. We sample 5 subsequent moves, which should be legal under the “make-believe” board. We verify that the modified board is valid and has at least one legal move before intervening.

Table 2 bottom-right The original model tries to move the deleted piece, serving as an intervention effectiveness baseline.

Table 2 top-left 92% of the time a non-ablated piece is selected. Most errors are attempts to move the ablated piece. The 85.4% quadrant errors are from trying to move a piece into the "empty" ablated square, which is not empty on the original board.

Addressing Other Questions

• We represent squares as (Mine, Yours, Empty) per Nanda+2023, training separate probe groups from each player's perspective. We will clarify this.
• Detailed analysis of Elo intervention impact would be valuable but is not possible during the discussion phase due to conference review guidelines.
• We did not measure per-move prediction accuracy. The character-level tokenizer loss cannot be straightforwardly converted to move accuracy.