Interpreting Arithmetic Reasoning in Large Language Models using Game-Theoretic Interactions

Thank you for your great efforts on the review of this paper. We will try our best to answer all your questions.

$\color{blue}Q1:$ "The paper did not study effects of attention heads separately. Given that the attention is primarily responsible for the token-token interaction, studying the cosine similarity of the head outputs could be illuminating."

A: Thank you. This is a valuable question. Following your advice, we conducted new experiments to study effects of attention heads separately. Specifically, we used attention scores of the last token to compute $v^{(l)}$ and then calculated the cosine similarity using this attention vector.

We conducted experiments on all 32 (0-31) attention heads of the MathCoder-L-7B model and observed consistent results across all of them: the model gradually focus more on operand-operator interactions (Table 1) as well as higher-order interactions (7th-order and 8th-order in Table 2), which aligns with our Insight 1. An example result for head 15 is shown as follows.

Table 1: Focality across different types of interactions at Layer $l$ using head 15 outputs. (Mathcoder-L-7B)

Table 2: Focality across interactions of different orders at Layer $l$ using head 15 outputs. (Mathcoder-L-7B template 1)

$\color{blue}Q2:$ "The score is averaged in a manner such that anti-parallel and parallel vectors are given equal importance. It would be interesting to look at histogram or higher moments to understand if such weightage is appropriate."

A: Thank you. A interesting question! To further investigate whether using the absolute value is appropriate, we analyzed the distribution of interaction effects. Specifically, we computed the skewness (with zero as the axis of symmetry) of interaction effects at each layer. The results (Table 3) show that the maximum skewness is 0.033 (< 0.05), indicating that the distribution does not exhibit a strong bias toward either the positive or negative side. Therefore, treating positive and negative values with equal importance (i.e., using the absolute values) is reasonable.

Table 3: Skewness (centered at 0) of interaction effects (MathCoder-CL-7B template 3).

$\color{blue}Q3:$ "What is the motivation behind the cosine similarity as the choice for $v^{(l)}$ , as opposed to say the mean squared difference?"

A: Thank you for the thoughtful question. We chose cosine similarity for computing $v^{(l)}$ because it effectively captures the directional alignment between hidden representations. In Transformer-based architectures, semantic similarity is often encoded in the direction of vector representations rather than their magnitude. This makes cosine similarity a natural and widely adopted choice, not only in language modeling, but also in other domains such as recommender systems, where it has proven effective in measuring similarity between user and item embeddings.

On the other hand, mean squared difference (MSE) measures absolute deviation in Euclidean space. While useful in some contexts, it conflates directional changes with scale differences, which may not be meaningful in the high-dimensional embedding spaces of LLMs.

Nevertheless, we conducted new experiments using the mean squared difference. We replaced the cosine similarity with the mean squared difference, and normalized it to the range $[0,1]$:

$v^{(l)}(x_T) = 1 - (MSE(f^{(l)}(x_T), f^{(l)}(x_N)) - d_{min}) / (d_{max} - d_{min} + ε)$

The experimental results confirm our main conclusion: For simple one-operator arithmetic problems, LLMs that perform well (i.e., LLaMA-2-7B) tend to focus more on operand-operator interactions (Table 4) and high-order interactions (Table 5). While LLMs with weak performance (i.e., GPT-J-6B) tend to focus more on background interactions (Table 6) and extremely low-order interactions (Table 7).

While MSE and cosine similarity yield similar conclusions, we tend to adopt cosine similarity in this paper to avoid the impact of feature scale differences.

Table 4: Focality across different types of interactions at Layer $l$ using MSE. (LLaMA-2-7B)

Table 5: Focality across interactions of different orders at Layer $l$ using MSE. (LLaMA-2-7B template 1)

Table 6: Focality across different types of interactions at Layer $l$ using MSE. (GPT-J-6B)

Table 7: Focality across interactions of different orders at Layer $l$​ using MSE. (GPT-J-6B template 1)

$\color{blue}Q4:$ "In Sec 4.1 the operand interactions do not seem to show any consistent trends. What is the consequence of this behavior?"

A: Thank you. That's a great question. Among the five LLMs that perform well on one-operator arithmetic problems (as shown in Table 8), the LLaMA-2-7B, Llemma-7B, and MathCoder-L-7B models show increasing focus on operand interactions in deep layers, while the CodeLlama-13B and MathCoder-CL-7B models do not show this trend. On the other hand, the CodeLlama-13B and MathCoder-CL-7B models show a more significant encoding of operand-operator interactions compared to the other three models. Among five LLMs, the LLaMA-2-7B model shows the weakest encoding of operand-operator interactions.

The Llemma-7B and MathCoder-L-7B models achieve the highest accuracy on one-operator arithmetic tasks (Table 8), suggesting that their strong focus on operand interactions is beneficial for arithmetic reasoning. For the LLaMA-2-7B model, we believe that its limitation in encoding operand-operator interactions is mitigated by its stronger focus on operand interactions.

As for the CodeLlama-13B and MathCoder-CL-7B model, although they do not tend to focus more on operand interactions, their strong modeling of operand-operator interactions is sufficient to support accurate arithmetic reasoning.

Table 8: Accuracy (%) of LLMs on one-operator arithmetic tasks.

$\color{blue}Q5:$ "Were embeddings considered at all? They could serve as a basic sanity check, since the tokens are not supposed to interact at the level of embeddings."

A: Thank you. This is an excellent question. As you correctly pointed out, tokens are not expected to interact at the embedding layer. Indeed, we verify this by computing interaction effects at the embedding layer, and all values turn out to be zero.

$\color{blue}Q6:$ "It was not clear to me what the message of Appendix E was."

A: Thank you for pointing this out. Appendix E aims to justify our choice of using the last token's embedding to compute $v^{(l)}$, that is, the cosine similarity between embeddings of the original input and that of the masked input.

To further illustrate this, we conducted the same experiment on the LLaMA-2-7B model with the prompt "How much is 5 plus 1? Answer is ." Results in Table 9 show that, except for the final token of the complete sentence, that is, token 10 ('?') and token 13 (the last token '▁'), the $v^{(l)}$ is nearly identical (close to 1). Only the embedding of the last token shows a significant change. Therefore, to simplify computation, we only focus on the change in the embedding of the last token, while ignoring the negligible changes in the embeddings of other tokens.

Moreover, several prior studies [Wendler et al. 2024] [Zhang et al. 2024] claim that in the Transformer architecture, the representation of last token includes information from all previous tokens.

Table 9: $v^{(l)}$ at different token positions across layers for the masked input "How much is 5 [mask] [mask]? Answer is ."

Thank you for your great efforts on the review of this paper. We will try our best to answer all your questions.

$\color{blue}Q1:$ "The analysis of interactions in intermediate layers relies solely on the embedding of the last input token is a simplification for complex transformer architectures. Might this choice lead to an incomplete or potentially biased understanding?"

A: Thank you. We have conducted experiments and found that the variation in middle-layer representations is primarily concentrated at the last token. Please see Appendix E for detailed results and analysis. Therefore, to simplify computation, we only focus on the change in the embedding of the last token, while ignoring the negligible changes in the embeddings of other tokens.

Nevertheless, we show an example to support the above claim. We tested a sample using the LLaMA-2-7B model: "How much is 5 plus 1? Answer is". This example contains 13 tokens: ['<s>', '▁How', '▁much', '▁is', '▁', '3', '▁times', '▁', '1', '?', '▁Answer', '▁is', '▁'].

We focused on a masked sentence: "How much is 5 [mask] [mask]? Answer is ," Table 1 reports the cosine similarity between embeddings of the original input and that of the masked input ($v^{(l)}$). Results in Table 1 show that, except for the final token of the complete sentence, that is, token 10 ('?') and token 13 (the last token '▁'), the cosine similarity between the embeddings of the original input and the masked input is nearly identical (close to 1). Only the embedding of the last token shows a significant change.

Moreover, several prior studies [Wendler et al. 2024] [Zhang et al. 2024] claim that in the Transformer architecture, the representation of last token includes information from all previous tokens.

Table 1: $v^{(l)}$ at different token positions across layers for the masked input "How much is 5 [mask] [mask]? Answer is ,".

$\color{blue}Q2:$ "The paper acknowledges the absence of unified masking strategies and employs specific tokens may influence the model's internal representations and the resulting interaction values."

A: Thank you. A good question. We apply masking to remove the semantic meaning of the corresponding token, thereby effectively blocking its influence on the LLM's output.

Since not all LLMs provide a [pad_token], we use the model's [unk_token] as a placeholder, which represents an out-of-vocabulary token. At present, different LLMs offer different [unk_token] without specific semantic meaning, so we adopt different placeholders for different models accordingly.

In recent years, some studies have proposed learning an optimal masking strategy for each model through training. However, such approaches are time-consuming, and thus we have not explored them in this paper. Currently, there is no unified standard for masking, and we also highlight this point in Appendix A.

$\color{blue}Q3:$ "The study focuses exclusively on "simple arithmetic problems", has not extended research to "more complex math word problems". This limits the generalizability of the findings."

A: Thank you. We conducted new experiments on a more complex math word problem: "Henry has 1 notebook, then Henry receives 8 notebooks. What's the total number of notebooks that Henry has?" This prompt is essentially an extension of a simple one-operator arithmetic problem "1+8=," but contains richer contextual information.

We categorize input variables into 4 types: numbers (i,e., "8", "1"), operators (i.e., "receives", "the total number of"), entity words (including names and objects, i.e., "Henry", "notebook", "notebooks"), and background words (all words other than the above three categories).

Following the same setting as in our paper, we also define four types of interactions:

1. Operand interactions: contain only numbers (no operators)

2. Operator interactions: contain only operators (no numbers)

3. Operand-operator interactions: contain both numbers and operators

4. Background interactions: contain neither numbers nor operators

We conducted experiments on the Mathcoder-L-7B model. The results confirm our main conclusion (Insight 1): The internal mechanism of LLMs for solving simple one-operator arithmetic problems is their capability to encode operand-operator interactions (Table 2) and high-order interactions (Table 3) during the forward propagation.

Table 2: Focality across different types of interactions at Layer $l$ on a free‐form math problem. (Mathcoder-L-7B)

Table 3: Focality across interactions of different orders at Layer $l$ on a free‐form math problem. (Mathcoder-L-7B)

$\color{blue}Q4:$ "The "focality" metric (Eq 8, 9) sums absolute interaction values. This approach may disregards the direction (positive or negative contribution) of interactions, which is crucial for understanding the precise nature of the model's reasoning."

A: Thank you. A good question! We conducted new experiments by dividing interactions into positive and negative categories.

For different types of interactions:

1. Positive value: $R_{+}^{(l)}(\Omega^{\text{type}}) = \frac{\mathbb{E}_{S \in \Omega^{\text{type}},\ I_S^{(l)} > 0} I_S^{(l)}}{Z^{(l)}}$ (Table 4)
2. Negative value: $R_{-}^{(l)}(\Omega^{\text{type}}) = \frac{\mathbb{E}_{S \in \Omega^{\text{type}},\ I_S^{(l)} < 0} I_S^{(l)}}{Z^{(l)}}$ (Table 5)

For interactions of different orders:

1. Positive value: $\kappa_{m,+}^{(l)} = \frac{\mathbb{E}_{|S|=m,\ I_S^{(l)} > 0} I_S^{(l)}}{Z^{(l)}}$ (Table 6)
2. Negative value: $\kappa_{m,-}^{(l)} = \frac{\mathbb{E}_{|S|=m,\ I_S^{(l)} < 0} I_S^{(l)}}{Z^{(l)}}$ (Table 7)

Results show that, in the later stage of forward propagation, the LLM shows the highest focus on operand-operator interactions (Table 4 and Table 5) and high-order interactions (8th-order and 9th-order in Table 6 and Table 7). The results confirm our main conclusion (Insight 1): The internal mechanism of LLMs for solving simple one-operator arithmetic problems is their capability to encode operand-operator interactions and high-order interactions during the forward propagation.

In summary, regardless of being positive or negative, interactions with large absolute values are considered to have a significant impact on the model’s output (i.e., whether pushing the prediction toward or away from the target answer, they still indicate strong influence).

Table 4: Focality across different types of interactions at Layer $l$ – Positive values. (Mathcoder-CL-7b)

Table 5: Focality across different types of interactions at Layer $l$ – Negative values. (Mathcoder-CL-7b)

Table 6: Focality across interactions of different orders at Layer $l$ – Positive values. (Mathcoder-CL-7b template 4).

Table 7: Focality across interactions of different orders at Layer $l$ – Negative values. (Mathcoder-CL-7b template 4).

Thank you for your great efforts on the review of this paper. We will try our best to answer all your questions.

$\color{blue}Q1:$ "Source code is not opened."

A: Thank you. Due to the conference policy, we cannot provide any links during the rebuttal phase. The code will be released publicly upon acceptance.

$\color{blue}Q2:$ Ask for verify the causal necessity. "The interaction effect can fit the output, but since this is an interpretability study, whether the causal necessity can be verified?"

A: Thank you. This is a very good question. Causal necessity means that a certain factor is necessary for the model's prediction, i.e., without this factor, the model would not be able to produce the same output. In our paper, the causal necessity is reflected in the influence of input variables on both the interaction effects and the LLM outputs.

In our paper, given an input $\boldsymbol{x}$ and an LLM, we can construct a logical model based on interactions to accurately fit the output of the LLM. The output of the logical model can be decomposed into the sum of the effects of all interactions in $\boldsymbol{x}$.

Given an LLM, changing the input $\boldsymbol{x}$ leads to changes in the LLM's output. This process reflects the causal necessity of input variables, while interactions are the outcome derived from the above causal analysis. However, we cannot directly change interaction effects to influence the LLM's output.

We conducted new experiments to demonstrate the causal necessity of the input variable (Table 1). Specifically, we input two prompts into the Mathcoder-L-7B model that differ only in the operator ("+" vs. "×"). We observe that both the model’s output and the focality on different types of interactions change accordingly.

Table 1: Changes in model output and focality across different types of interactions at Layer $l$ when flipping the operator in arithmetic prompts. (Mathcoder-L-7B)

Prompt: "How much is 2 plus 4? Answer is " Model output: "6"

Prompt: "How much is 2 times 4? Answer is " Model output: "8"

$\color{blue}Q3:$ Ask for new experiments on free-form math problems. "Experiments use templated arithmetic questions. It’s unclear how it performs in free‐form math problems."

A: Thank you. A good question. We have conducted new experiments on a free-form math problem : "Henry has 1 notebook, then Henry receives 8 notebooks. What's the total number of notebooks that Henry has?" We categorize input variables into 4 types: numbers (i,e., "8", "1"), operators (i.e., "receives", "the total number of"), entity words (including names and objects, i.e., "Henry", "notebook", "notebooks"), and background words (all words other than the above three categories).

Following the same setting as in our paper, we also define four types of interactions:

1. Operand interactions: contain only numbers (no operators)

2. Operator interactions: contain only operators (no numbers)

3. Operand-operator interactions: contain both numbers and operators

4. Background interactions: contain neither numbers nor operators

We conducted experiments on the Mathcoder-L-7B model. We observe that during the forward propagation, the LLM’s focus on operand-operator interactions (see Table 2) gradually increases, along with its focus on higher-order interactions (7th-order and 8th-order in Table 3). This is consistent with the Insight 1 in our paper. This experiment demonstrates that our method can be extended to free-form math problems.

Table 2: Focality across different types of interactions at Layer $l$ on a free‐form math problem. (Mathcoder-L-7B)

Table 3: Focality across interactions of different orders at Layer $l$ on a free‐form math problem. (Mathcoder-L-7B)

Thank you for your great efforts on the review of this paper. We will try our best to answer all your questions.

$\color{blue}Q1:$ "The paper's empirical analysis is based on several assumptions which are too strong to be fully convincing. The authors use cosine similarity (Eq.3) to quantify hidden-state changes across layers, which missed large magnitude shifts in a single dimension. Without metrics, like Euclidean distance, the analysis risks overlooking substantial activation shifts."

A: Thank you. We would like to clarify whether the reviewer’s concern — "they are several assumptions which are too strong to be fully convincing" — refers specifically to our use of cosine similarity in Eq.3. If so, we conducted new experiments using Euclidean distance.

We replaced the cosine similarity in the original Eq.3 with Euclidean distance, and normalized it to the range $[0,1]$:

$v^{(l)}(x_T) = 1 - (EuclideanDistance(f^{(l)}(x_T), f^{(l)}(x_N)) - d_{min}) / (d_{max} - d_{min} + ε)$

The experimental results confirm our main conclusion: For simple one-operator arithmetic problems, LLMs that perform well (i.e., the LLaMA-2-7B model) tend to focus more on operand-operator interactions (Table 1) and high-order interactions (Table 2) during forward propagation. While LLMs with weak performance (i.e., the GPT-J-6B model) tend to focus more on background interactions (Table 3) and extremely low-order interactions (Table 4).

Table 1: Focality across different types of interactions at Layer $l$ using Euclidean distance (LLaMA-2-7B).

Table 2: Focality across interactions of different orders at Layer $l$ using Euclidean distance (LLaMA-2-7B template 1).

Table 3: Focality across different types of interactions at Layer $l$ using Euclidean distance (GPT-J-6B).

Table 4: Focality across interactions of different orders at Layer $l$ using Euclidean distance (GPT-J-6B template 1).

$\color{blue}Q2:$ "Harsanyi interactions require summing over 2^n subsets, which is infeasible when input length grows beyond a handful of tokens."

A: Thank you. A good question! When the input length grows beyond a handful of tokens, the computational cost of interaction analysis indeed rises.

To deal with this issue, we can adopt the following approaches.

1. Analyzing informative variables while treating uninformative variables as background to reduce computational cost while preserving core semantic information. Previous research [Chen et al., 2024] [Ren et al., 2024] [cite 1] has demonstrated that selecting informative input variables does not fundamentally compromise the faithfulness of interactions.
2. Phrase-level aggregation, where related tokens are merged into meaningful phrases without compromising semantic integrity, can reduce the computational cost.

We have conducted new experiments using the two approaches described above. Specifically, we extended a simple one-operator arithmetic problem "1+8=" into a longer prompt: "Henry has 1 notebook, then Henry receives 8 notebooks. What's the total number of notebooks that Henry has?" We categorize input variables into 3 types: numbers (i,e., "8", "1"), operators (i.e., "receives", "the total number of"), and entity words (including names and objects, i.e., "Henry", "notebook", "notebooks"). By using Approach 1, all other words not belonging to the above three categories are treated as background. By using Approach 2, the phrase "the total number of" is grouped as a single variable.

Following the same setting as in our paper, we also define four types of interactions:

1. Operand interactions: contain only numbers.

2. Operator interactions: contain only operators.

3. Operand-operator interactions: contain both numbers and operators.

4. Background interactions: contain neither numbers nor operators.

We conducted experiments on the Mathcoder-L-7B model. The results confirm our Insight 1: The internal mechanism of LLMs for solving simple one-operator arithmetic problems is their capability to encode operand-operator interactions (Table 5) and high-order interactions (Table 6) during the forward propagation.

Nevertheless, we acknowledge that when $n$ is too large, the computation becomes infeasible. In fact, we have discussed this issue in the Limitation section of our paper.

Table 5: Focality across different types of interactions at Layer $l$ on a free‐form math problem. (Mathcoder-L-7B)

Table 6: Focality across interactions of different orders at Layer $l$ on a free‐form math problem. (Mathcoder-L-7B)

[cite 1] Li M. et al. Does a neural network really encode symbolic concepts? ICML, 2023.

$\color{blue}Q3:$ "This paper does not compare its interaction decomposition with other interpretability or attribution techniques."

A: Thank you. A good question! Traditional interpretability and attribution methods, such as Grad-CAM and Shapley values, often lack rigorous theoretical guarantees. In contrast, the interactions has been proven to be faithful explanations by a series of theoretical guarantees [Ren et al., 2023a]. The faithfulness of using interactions to explain an LLM is reflect as follows.

1. The output score of the LLM on any input sentence can be represented as the sum of the effects of all interactions.
2. The output score of the LLM on any masked sentence can be represented as the sum of the effects of all triggered interactions.

Moreover, most attribution methods focus on the influence of individual input variable. However, a DNN does not independently use each variable for inference. Instead, a DNN relies on interactions among input variables for inference. For example, the interaction between "green" and "hand" forms the concept of "beginner," a meaning that cannot be derived from either word alone.

Nevertheless, we conducted new experiments using the Shapley value-based method on the same single-operator templates. We computed Shapley values for different types of input variables during the forward propagation of the LLM. The results show that LLMs that perform well (i.e., the LLaMA-2-7B model) tend to focus most on operators, followed by operands, and lastly on background words (Table 7). However, the Shapley value-based method fail to capture the joint effect of operands and operators on the model’s reasoning process.

Table 7: Average Shapley values for different types of input variable across layers. (LLaMA-2-7B)

$\color{blue}Q4:$ "Can you provide a simple intervention study to produce the predicted jumps in interaction scores and output, thereby demonstrating causality? For example, flipping the operator token from ‘×’ to ‘+’."

A: Yes, we followed your advice to conduct experiments to demonstrate the causality of the input operator variable (Table 8). Specifically, we input two prompts into the Mathcoder-L-7B model that differ only in the operator ("+" vs. "×"). We observe that both the model’s output and the focality on different types of interactions change accordingly.

However, we cannot directly change the interaction effect to observe output changes, as it is an outcome of causal analysis based on how input changes lead to output differences, not a controllable variable in the process.

Table 8: Changes in model output and focality across different types of interactions at Layer $l$ when flipping the operator in arithmetic prompts. (Mathcoder-L-7B)

Prompt: "How much is 2 plus 4? Answer is " Model output: "6"

Prompt: "How much is 2 times 4? Answer is " Model output: "8"