Which Data Attributes Stimulate Math and Code Reasoning? An Investigation via Influence Functions

Thank you for your attentive comments and useful suggestions! We are glad you thought our paper had direct downstream implications for LLM fine-tuning. We address your concerns point by point below.

W1: The new method ("Infra") might have somewhat limited novelty. It seems very close to an implementation of previous influence approaches (specifically, EK-FAC; https://arxiv.org/abs/2308.03296, as cited in the paper).

Thank you for your feedback. We would like to emphasize that our main contribution lies in the novel application of influence functions to trace how training data during the SFT phase shapes LLM reasoning for the first time, presenting key findings with substantial practical value, as acknowledged by Reviewer mqmh. Reviewer AwiZ also noted that we 'target an important problem and uncover multiple interesting findings', and Reviewer x5z4 praised our work for 'the analysis fills a critical gap that needed to be addressed'.

Furthermore, unlike prior approaches, our work introduces a fine-grained analysis: sequence-level analysis reveals the counterintuitive insight that "exploration" behavior can enhance reasoning, while token-level analysis provides valuable insights into the interaction mechanisms between math and code training data.

W2 & Q2: It is somewhat expected that including more difficult problems in the last tuning phase would benefit performance. Could the tuning results on the proposed Difficulty Flipped dataset be because that data has substantially more difficult problems overall than all the other comparison datasets?

Indeed, one of our key findings is that high-difficulty math examples improve both math and code reasoning, and therefore, the performance improvements observed in the Difficulty Flipped dataset can partly be attributed to the inclusion of more difficult math problems.

The reviewer mentioned the intuition that difficult problems may lead to better performance. This can be wrong in practice, e.g., for the code capacities. To validate this, we conduct an ablation study on Qwen2.5-Instruct-7B, with both high-difficulty math and code samples for training. The results in the table below show a decline in coding performance compared to the proposed Difficulty Flipped dataset. This may be due to the fact that easier code problems, with their clearer structural formats, are more effective for enhancing code reasoning, especially when integrated with math CoTs that already provide strong reasoning skills.

Thank you for pointing this out, and we will add this ablation result to the revised version.

W3: Other works have applied influence functions to LLMs for reasoning and/or mathematical tasks (e.g. https://arxiv.org/abs/2411.12580, https://arxiv.org/abs/2410.17413). More fine-grained breakdowns could make the paper more informative, by isolating exactly why specific math and code examples influence one another.

Thanks for pointing out this valuable direction for deeper interpretability of cross-domain influence. First, we would like to clarify that these works primarily focus on attributing LLM performance to pretraining data, whereas our work targets the SFT phase. We will include citations to these works in the revised version.

Regarding more fine-grained breakdowns, as shown in our paper (Figure 3(a) and Figure 4), we find that symbolic math problem-answer pairs introducing or solving for x are among the most beneficial for code-related tasks. This may be because the process of reasoning with symbolic variables in math closely mirrors the patterns used in programming tasks.

To further explore long-CoT reasoning settings, we conduct a case study analyzing the top 20 math examples with the highest impact on LiveCodeBench performance. Our observations suggest that the most influential math problems are predominantly symbolic manipulation tasks requiring equation rearrangement and factorization (see some examples below), likely because they cultivate algebraic reasoning and variable isolation skills that are fundamental for implementing mathematical algorithms and debugging complex code expressions. We will include this in the revised version.

Question: If $b>1$, $x>0$, and $(2x)^{\\log_b 2}-(3x)^{\\log_b 3}=0$, then what is $x$?
Answer(partial): So, if I take \$ \\log_b \$ of both sides:
\$ \\log_b \\left( (2x)^{\\log_b 2} \\right) = \\log_b \\left( (3x)^{\\log_b 3} \\right) \$.
Using the logarithm power rule, \$ \\log_b (a^c) = c \\cdot \\log_b a \$, so applying that here, the equation becomes:
\\( \\log_b 2 \\cdot \\log_b (2x) = \\log_b 3 \\cdot \\log_b (3x)

Question: If $x, y, z$ are reals such that $0\\le x, y, z \\le 1$, show that $\\frac{x}{y + z + 1} + \\frac{y}{z + x + 1} + \\frac{z}{x + y +  1} \\le 1 - (1 - x)(1 - y)(1 - z)$
Answer(partial): Let's denote S = x + y + z, P = xy + yz + zx, Q = xyz. Then RHS is S - P + Q.
The LHS is sum of x\/(S - x + 1). Wait, because denominator for the term with x is y + z + 1 = (y + z) + 1 = (S - x) + 1. So LHS can be written as: Sum_{x} [x \/ (S - x + 1)]. 

Q1: Is there a way to distinguish whether the observed influence scores are primarily driven by surface-level similarities between examples, vs. more abstract "procedural" similarities (e.g. https://arxiv.org/abs/2411.12580)?

Thank you for raising this insightful question. To assess whether influence scores are primarily driven by surface-level similarities, we compute BERTScore[1] between each of the 100 AIME and MATH500 examples and their top-10 most influential training examples. BERTScore computes semantic similarity between tokens in a candidate sentence and a reference sentence using contextual embeddings, making it suitable for assessing surface-level overlap.

We report the maximum BERTScore(F1) among the top-10 influential examples for each target question (across 100 total targets) in the table below. Most maximum scores are around 0.65, suggesting that the influence scores are unlikely to be dominated by superficial lexical overlap, but more likely reflect abstract procedural patterns--consistent with findings in https://arxiv.org/abs/2411.12580.

[1] BERTScore: Evaluating Text Generation with BERT, ICLR 2020

Q3: Is there evidence that the high-difficulty math examples are highly influential because they contain important reasoning strategies that the model relies on for other math and code reasoning problems? Are there ways to characterize or classify these different reasoning strategies? More discussion and results on this point would help clarify exactly why these types of examples are so important.

Thank you for this interesting question for the deeper interpretation of why high-difficulty math examples are highly influential. To investigate this, we compare 100 high-difficulty examples with top influence scores and 100 low-difficulty examples with top influence scores. We observe that high‑difficulty samples consistently exhibit (1) Knowledge Integration, where solutions draw on concepts from multiple domains, and (2) Strategy Diversity, where diverse reasoning patterns are employed within a single solution. We use GPT-4o based annotation (prompt templates are shown below) to quantify these phenomena. As shown in the table below, both knowledge integration and strategy heterogeneity occur significantly more often in high‑difficulty examples, which likely underlies their better transferability and generalization to other reasoning problems.

**Task Description**
You will be given a piece of text. Please evaluate whether this text exhibits knowledge integration. Knowledge integration occurs when multiple domains or disciplines of knowledge and strategies are combined to solve a problem effectively—for example, blending algebra, geometry, and probability in a single math solution. Look specifically for instances where the text weaves together concepts from different fields.
**Examples**
“To solve the variance problem, we first wrote down the expectation and second-moment formulas using probability theory, then applied algebraic manipulation to solve the resulting quadratic equation for p.”
**Output Format**
## Thoughts
[Briefly note what you observed that indicates knowledge integration, and which disciplines or techniques are being combined.]

## Does knowledge integration occur?
[yes / no]

**Task Description**
You will be given a piece of text. Please evaluate whether this text exhibits strategy diversity. Strategy diversity means using three or more distinct problem-solving methods or reasoning paths—such as decomposition, recursion, dynamic programming, greedy selection, etc.—within the same solution. Focus on spotting multiple, clearly different strategies being applied in tandem or in sequence.
**Examples**
“First we apply a divide-and-conquer approach to break the problem into subproblems, then we optimize each subproblem via dynamic programming.”
**Output Format**
## Thoughts
[Briefly note what you observed that indicates strategy diversity, and which disciplines or techniques are being combined.]

## Does strategy diversity occur?
[yes / no]

Given these improvements and additional experiments to your initial concerns, we hope you would like to raise your score.

Thank you for the positive feedback and useful suggestions! We address your concerns point by point below.

W1: The conclusions of this work are limited to the SFT stage, while model capabilities largely stem from pre-training. Although this focus may be insufficient, the complexity and cost of analyzing pre-training make it understandable.

Thank you for your feedback. Indeed, it is widely acknowledged that the capabilities of LLMs are learned during pre‑training. However, it is widely recognized that the SFT phase is indispensable for unlocking and shaping those capabilities [1,2]. Thus, characterizing which properties of SFT data most effectively stimulate reasoning remains highly valuable. We acknowledge that investigating the pre-training phase would be illuminating, but it is beyond the scope of this work due to its complexity and scale; we therefore propose such an investigation as important future work.

[1] Towards Reasoning in Large Language Models: A Survey, ACL 2023

[2] Octothinker: Mid-training incentivizes reinforcement learning scaling. arXiv:2506.20512

W2 & Q3: The design of the influence scores lacks the necessary theoretical or empirical justification to support the reasonableness of such a formulation. What was the rationale behind the design of the influence scores?

The influence score formulation is grounded in well-established theoretical foundations. It originates from the work [1], which rigorously derives how the removal of a training example affects model parameters and, in turn, model predictions. This approach provides a principled framework for quantifying the impact of individual training samples on model behavior.

In our work, we design the influence scores by adopting the mean log-probabilities on the correct answers as the response function f, because this metric closely reflects the model's reasoning ability. We appreciate this question and will include these clarifications in the revised version.

[1] Understanding Black-box Predictions via Influence Functions, ICML 2017

W3 & Q1: This paper examines the impact of data on model capabilities solely from the perspective of training data → performance, lacking deeper investigation and understanding of the internal mechanisms of the model.

Our primary focus is indeed on understanding how training data influences model performance, which we believe is both scientifically meaningful and practically valuable—especially for guiding data curation and mixture strategies. Reviewer AwiZ also noted that we 'target an important problem and uncover multiple interesting findings'.

While we agree that probing the internal mechanisms of LLMs is important, it remains a highly challenging task due to the complexity and scale of these models. Nevertheless, our influence-based approach provides a potential pathway for such analysis. For instance, by decomposing the gradients involved in the influence computation, one could potentially attribute specific layers or modules within the model to reasoning performance. We consider this an interesting direction and leave it as future work.

Q2: Are there any additional key experiments or findings that should have been included in the conclusion?

We would like to clarify that the current conclusion section has systematically summarized our key findings:

• Cross-domain examples—particularly high-difficulty math and low-difficulty code—enhance reasoning performance across domains.
• Sequence-level analysis reveals that exploratory behaviors in long CoT consistently improve performance, challenging assumptions that such behaviors are merely overthinking.
• Token-level analysis further uncovers domain-specific reasoning patterns between math and code.

Q4: Does replacing the Hessian matrix with the EK-FAC method compromise the reliability of the conclusions? Have the authors compared the impact of other alternatives on the final results?

Thanks for pointing this out. Computing the exact Hessian is intractable for LLMs due to the prohibitive memory and computational costs. While K-FAC is a commonly used approximation, prior work has shown that it is less accurate than EK-FAC [1]. Moreover, EK-FAC has been shown to offer a good trade-off between computational efficiency and accuracy [1,2], which supports the reliability of our conclusions. Additionally, our SFT experiments (Table 1 & Table 2b) further validate the robustness of these findings.

[1] Fast Approximate Natural Gradient Descent in a Kronecker-factored Eigenbasis, NIPS 2018

[2] Studying Large Language Model Generalization with Influence Functions, arXiv:2308.03296

Thank you for the feedback and useful suggestions! We are glad you thought our paper 'targeted an important problem and had multiple interesting findings'. We address your concerns point by point below.

W1 & Q1: Lack of comparison (experimentally or just discussion) with data selection methods.

Thank you for your feedback. We would like to emphasize that our work is not for data selection, which seeks a minimal subset of training examples for efficient domain transfer. Rather, our primary goal is different: we employ influence functions to attribute model performance back to individual training examples, thereby revealing which data types drive cross-domain reasoning improvements, which can in turn inform the design of more effective data curation and mixture strategies.

Moreover, influence-based sequence/token-level analysis enables us to ask why certain samples or patterns are helpful—e.g., identifying specific reasoning patterns that correlate with high influence—which is difficult to achieve with data selection methods like LESS[1] and S2L[2]. We argue that this analytical perspective offers complementary insights that could help guide future improvements in training data design.

[1] LESS: Selecting Influential Data for Targeted Instruction Tuning. ICML 24

[2] SmallToLarge (S2L): Scalable Data Selection for Fine-tuning Large Language Models by Summarizing Training Trajectories of Small Models. NeurIPS 24

W2 & Q2: The influence function mainly relies on EK-FAC, which is not novel. The limitation of EK-FAC needs to be discussed.

Thanks for pointing this out. Computing the exact Hessian is intractable for LLMs due to prohibitive memory and computational requirements. Among available approximations, EK-FAC [1] has been shown to provide a good trade-off between computational efficiency and accuracy. Nevertheless, we acknowledge its limitations and will add this discussion to the revised version.

• Block-Diagonal and Kronecker Factorization Assumptions. It assumes that different layers are independent, ignoring inter-layer curvature interactions. Moreover, it assumes each layer’s Fisher block is exactly separable as a Kronecker product, which may poorly capture complex curvature.
• Increased computation overhead than K-FAC. EK‑FAC incurs approximately twice the computational cost compared to K‑FAC by forming and inverting two Kronecker factors and performing their eigendecompositions.

[1] Fast Approximate Natural Gradient Descent in a Kronecker-factored Eigenbasis, NIPS 2018

W3 & Q3: As a study paper, the authors need to run the experiments multiple times and then conduct a statistical analysis for a rigorous results report.

We would like to clarify that the influence‑score computation (Equation 3) is entirely deterministic, and thus there is no randomness in the results related to influence scores. For SFT results in Tables 1 and 2(b), we retrain models with two additional random seeds and report the resulting standard deviations (shown in parentheses) for robustness. These new results will be included in the revised version. Thank you for your suggestion.

Given these clarifications and improvements to your initial concerns, we hope for your reconsideration in raising your score.

Thank you for the positive feedback and useful suggestions! We are glad you think our analysis is important in that it fills a gap that needed to be filled. We address your concerns point by point below.

Weakness: Method-wise, my understanding is that there was not significant novel innovation.

Thank you for your feedback. We would like to emphasize that our main contribution lies in the novel application of influence functions to trace how training data during the SFT phase shapes LLM reasoning for the first time, presenting key findings with substantial practical value, as acknowledged by Reviewer mqmh. Reviewer AwiZ also notes that we 'target an important problem and uncover multiple interesting findings'.

Q1: What was the reason that token-level attribution used gradients, whereas sequence-level attribution used ablation, and not just the sum of token-level attributions from gradients?

Summing token-level attributions from gradients only treats the sequence in its role as a prediction target and ignores its function as input context (see Equation 7) due to the AR nature. In contrast, sequence‑level attribution via ablation quantifies the effect of removing the sequence from training data, thereby capturing its contribution both as input and as target. Thank you for this question, and we will add this clarification to the revised version.

Q2: Was normalizing the influence scores so that magnitude of the gradient is ignored, i.e. RelatIF, considered when doing the analysis? I wonder if sequence length would correlate with influence, if there is not normalization.

Thank you for raising this interesting question. First, we would like to clarify that influence scores, as defined in Equation  3, measure the effect on the model performance when a training sample is removed; accordingly, we do not normalize by gradient norm or sequence length, since the raw gradient magnitude itself may carry information about a sample’s impact.

Furthermore, to investigate whether sequence length would correlate with influence, we compute the Spearman rank correlation between the original influence scores and those obtained after dividing by sequence length of 5000 samples, yielding ρ = 0.48. This indicates moderate correlation, which is consistent with findings in related works[1,2].

[1] The Impact of Reasoning Step Length on Large Language Models. https://arxiv.org/abs/2401.04925

[2] LIMO: Less is More for Reasoning. https://arxiv.org/abs/2502.03387

We hope these reclarifications and explanations of our method in response to your initial concerns can convince you to increase your score.