Enhancing Mathematical Reasoning in Language Models Through Focused Differentiation Training

Thank you for your thorough and positive review. The key difference between DPO and FDT lies in their gradient update rules: Equation 7 (Traditional DPO):

$$\begin{aligned} \Delta W[y]=c(x, y_w, y) h(x, y_w^{1:k-1})^\top+c(x, y_l, y) h(x, y_l^{1:k-1})^\top, \end{aligned}$$

Equation 8 (FDT):

$$\begin{aligned} \Delta W[y]=\alpha(c(x, y_w, y)-c(x, y_l, y))(h(x, y_w^{1:k-1})-h(x, y_l^{1:k-1}))^\top, \end{aligned}$$

During fine-tuning, the pre-trained weights already handle shared semantic components effectively. FDT explicitly focuses on distinctive features through its difference-based update rule, while DPO must implicitly learn to distinguish these components. This targeted approach is particularly important for mathematical reasoning, where critical differences often exist within similar contexts. Our theoretical results (Theorem 2) provide formal guarantees that FDT bounds the influence of shared components while amplifying distinctive features, leading to more efficient optimization of discriminative aspects.

We thank the reviewer for their detailed feedback. We address each main concern below:

The relationship between Eq 7 and Eq 8

We would like to clarify that these two equations serve different purposes and are not meant to be mathematically equivalent. Equation 7 shows the standard DPO gradient update, which represents how DPO traditionally updates the weight matrix by considering both correct and incorrect responses independently. Equation 8 presents our proposed FDT update rule, which is intentionally designed to be different from Equation 7, as it explicitly focuses on amplifying the differences between correct and incorrect responses. Rather than treating them independently (as in Equation 7).

The assumption in Theorem 2 indicates that the hidden states for correct and incorrect answers should be very close.

We would like to clarify an important misunderstanding about our assumptions and methodology. Our assumption is not that "hidden states for correct and incorrect answers should be very close." Rather, we only assume similarity in their norms ($|\mu_+|^2-|\mu_-|^2\leq \delta$), not in their vector representations. We have strong empirical evidence supporting this assumption. Our analysis shows that the relative differences in hidden state norms between correct and incorrect responses are consistently small in Appendix B, Figure 3. This figure illustrates the distribution of these relative differences across all sample pairs for two different models: Qwen2.5-3B-Instruct and Mistral-7B-Instruct-v0.3. The histograms reveal that the relative differences are predominantly concentrated around zero for both models. Qwen2.5-3B-Instruct exhibits a mean of 0.0321 and a median of 0.0220, while Mistral-7B-Instruct-v0.3 shows even smaller differences with a mean of 0.0100 and a median of 0.0081. These consistently small differences suggest that the samples maintain similar representation norms in the models' hidden state spaces, regardless of their chosen or rejected labels.

In addition, the similar and different aspects between correct and incorrect answers are difficult to be decoupled. The method in this paper does not provide any effective strategies for decoupling them, but rather assumes that they can be decoupled directly.

<!-- There have been several recent works that have shown that the hidden states of correct and incorrect answers can be decomposed into two parts [1,2]. -->

Recent research provides strong evidence that hidden states differences can effectively decouple semantic features:

1. Turner et al. (2024) demonstrate that taking differences between hidden states is an effective way to isolate specific semantic components. They show that their "steering vectors" (computed as activation differences) can selectively modify specific semantic features while preserving others [1]. For example, their sentiment steering vector successfully shifts text sentiment without affecting other content - directly evidencing semantic decoupling. Their comparison with random vectors shows that these differences capture targeted semantic changes rather than broad distributional shifts.
2. Marks and Tegmark (2024) provide additional evidence that hidden state differences in language models effectively isolate semantic components [2]. Their experiments reveal that "the model computed and linearly represented some feature which correlates with truth on both cities and neg_cities but with opposite signs" (Section 4.1). This demonstrates that taking differences between hidden states can extract specific semantic features (like truthfulness) while preserving other aspects.

Similar conclusions have been drawn in other works [3,4].

the experiments do not provide valid ablation analysis to demonstrate the effectiveness of the proposed modules

1. Regarding ablation studies: Our method is designed as a plug-in enhancement that can be integrated into any pairwise preference optimization framework. The baseline methods (DPO, Step-DPO) effectively serve as ablation studies - they are identical to our approach but without the FDT component. The performance improvements over these baselines directly demonstrate FDT's effectiveness.

2. Concerning significance of improvements: As another reviewer noted, while not groundbreaking, our method provides "a very useful gadget for doing math reasoning" that can be widely applied. The consistent improvements across multiple models (Llama-3.2-3B-Instruct and Qwen2.5-3B-Instruct) and different base methods (DPO, Step-DPO) demonstrate the robustness of our approach.

We appreciate the reviewer's thoughtful comments about the need for clearer motivation and analysis of our approach. We address these concerns by providing additional explanations and evidence:

1. Theoretical Foundation for Hidden State Differences

   Our approach of focusing on hidden state differences is well-grounded in recent research on semantic feature isolation. Turner et al. (2024) demonstrated that differences between hidden states effectively isolate specific semantic components while preserving others. Their work shows that computing differences between hidden states can selectively modify specific semantic features without affecting other content - providing direct evidence for semantic decoupling. This aligns with our method's goal of amplifying the distinctive semantic components that differentiate correct from incorrect mathematical reasoning.

2. Mechanism of Semantic Differentiation

   The key distinction between traditional DPO (Equation 7) and our FDT approach (Equation 8) lies in how they handle semantic information:

   Equation 7 (Traditional DPO):

   $$\begin{aligned} \Delta W[y]=c(x, y_w, y) h(x, y_w^{1:k-1})^\top+c(x, y_l, y) h(x, y_l^{1:k-1})^\top, \end{aligned}$$

   Equation 8 (FDT):

   $$\begin{aligned} \Delta W[y]&=\alpha(c(x, y_w, y)-c(x, y_l, y))(h(x, y_w^{1:k-1})-h(x, y_l^{1:k-1}))^\top, \end{aligned}$$

   While DPO treats correct and incorrect responses independently, FDT explicitly computes their difference to focus on discriminative features. This is not merely a mathematical transformation - it fundamentally changes what the model learns to attend to.

3. Last Layer Focus and Theoretical Guarantees

   We focus on the last layer's hidden states because they directly influence the model's logits through the output layer weight matrix W. This direct connection means that modifying the last layer's representations has the most immediate impact on the model's predictions. Our theoretical results formalize how this modification works:

• Theorem 2 guarantees that our update rule bounds the influence of shared semantic components: $\Delta W[y](h(y|x, y_w^{1:k-1})+h(y|x, y_l^{1:k-1}))\le 4\alpha\delta$. This means that features common to both correct and incorrect responses have limited impact on the model's predictions.

• Corollary 1 ensures that our method effectively amplifies the distinctive features: $\Delta W[y](h(y|x, y_w^{1:k-1})-h(y|x, y_l^{1:k-1}))\ge \frac{1}{2}\alpha\|\mu_+-\mu_-\|$. This theoretical guarantee shows that the features that distinguish correct from incorrect responses are emphasized in the model's decision-making process.

  Together, these results demonstrate that by focusing on the last layer, where hidden states directly map to predictions through W, we can effectively control what information influences the model's outputs - suppressing shared features while amplifying discriminative ones. While we directly modify the last layer's hidden states, the gradient backpropagation through the model ensures that earlier layers are also optimized to better capture and propagate discriminative features.

We sincerely thank the reviewer for their thorough and constructive feedback. We address each main concern below:

Experiments are not thorough or mature enough.

1. We have conducted additional experiments with more LLM variants including deepseek-llm-7b-chat and Mistral-7B-Instruct-v0.

2. Our method is designed as a plug-in enhancement that can be integrated into any pairwise preference optimization framework. The baseline methods (DPO, Step-DPO) effectively serve as ablation studies - they are identical to our approach but without the FDT component. The performance improvements over these baselines directly demonstrate FDT's effectiveness.
3. As another reviewer noted, while not groundbreaking, our method provides "a very useful gadget for doing math reasoning" that can be widely applied. The consistent improvements across multiple models (Llama-3.2-3B-Instruct and Qwen2.5-3B-Instruct) and different base methods (DPO, Step-DPO) demonstrate the robustness of our approach.

The reason why the entire setting in the paper was framed as a problem for mathematical reasoning alone

Mathematical reasoning presents an ideal test case for our method because it involves subtle semantic differences that can dramatically change the correctness of a response. In mathematical problem-solving:

1. Small semantic variations often lead to critically different outcomes. For example, the difference between "greater than" versus "greater than or equal to" can completely change a solution's validity. Our method specifically addresses these fine-grained semantic distinctions through focused differentiation of hidden states.
2. Mathematical reasoning provides clear ground truth for evaluating semantic differences. Unlike more subjective tasks, mathematical problems have definitive correct and incorrect answers, making them perfect for validating our approach to semantic differentiation.
3. The structured nature of mathematical language allows us to isolate and analyze how our method handles semantic differences in a controlled setting. This is particularly valuable for demonstrating the effectiveness of our hidden state differentiation approach.

Does the assumption in Theorem 2 require the hidden states to be bounded with some constant $M$ and $\delta$ for every input?

No, the assumption does not require the hidden states to be bounded with some constant $M$ and $\delta$ for every input. The assumption requires that the mean of the hidden states of correct and incorrect responses is bounded. We have provided empirical evidence supporting this assumption in Appendix B, Figure 3, which demonstrates that this assumption is generally valid across different models