Embedding Trajectory for Out-of-Distribution Detection in Mathematical Reasoning

Thanks for your constructive comments! We will respond to the weaknesses and questions you raised in the following areas:

More Metrics (W1 & Q3)

Thank you for suggesting richer evaluation metrics about AUPR and F1. The results are below:

On both metrics, our approach still maintains a significant lead.

Discussion about k (W3 & Q2)

Thanks for suggesting a more detailed discussion of $k$. In Section 5 and Appendix G.1, we have conducted ablation studies for cases of $k \leq 5$. To support our claim of choosing $k \leq 5$ and the evidence about the over-smoothing phenomenon when $k$ is too large, we report more AUROC results in Llama2-7B and GPT2-XL as we continue to increase $k$ values:

When $k>5$, the AUROC value on Llama2-7B shows a sharp drop and stabilizes around 50. The AUROC value on GPT2-XL remains high, but also shows a sharp decrease when $k$ increases close to 10. This is because GPT2-XL has 1.5 times more layers than Llama2-7B, the trajectory information is richer and the noise will be relatively more.

Overall, as $k$ continues to increase, the useful trajectory volatility information is gradually blurred even though more noise is erased, causing over-smoothing. Therefore, $k \leq 5$ is a better trade-off range, this is why we choose $k \leq 5$.

Experimental Setup about Figure 3 (Q4)

The ID data curve is the average of all samples in the MultiArith dataset, and the OOD data curve is the average of all samples in the five domains of the MATH dataset (Algebra, Geometry, Counting and Probability, Number Theory, and Precalculus). We will add the setup in the updated version.

Method Detail: Fixed Embedding Dimension (Q1)

The value of the embedding dimension is arbitrary and does not need to be fixed, but the embedding dimensions of neighboring layers should be equal, otherwise they cannot be subtracted.

For language models, the output dimension of the hidden layer is the embedding dimension set in the model configuration (e.g., 4096 for Llama2-7B and 1600 for GPT2-XL), so there is no problem of inconsistent dimensionality.

Paper Writing (W2 & Q5)

Thank you for pointing out the mistakes in our writing, we will correct them in the updated version.

We expect the above responses to address your concerns, and look forward to your more positive comments.

Thanks for your constructive comments! We will respond to your concerns one by one.

W1: The motivation is not strong and convincing. Figure 1 ...

1. Input Space

As for the phenomenon that embedding varies less in different domains of input space: We have discussed it in Section 5 and conducted experiments in Appendix G.2 to demonstrate this.

2. Output Space

Please refer to "General Rebuttal: Existence and Universality of "Pattern Collapse" phenomenon in mathematical reasoning" for detailed responses and evidences.

We must claim a key fact: For generative language models (GLMs), they model real numbers or mathematical expressions not in a mathematical sense, but based on discrete token sequences after tokenization. Thus, the collapse occurs at the token level, not at the full mathematical expression level. Due to the autoregressive generative nature of GLMs, the collapse phenomenon occurs during the prediction of each token.

We agree with your opinion that the number of values on the real line is infinite in a mathematical sense. However, after tokenization, they contain only 0-9 number tokens and a limited number of special symbols, such as decimal points, slashes, root signs. This means that two largely different expressions in the mathematical sense may cover many of the same tokens.

We emphasize two key conclusions from the statistic data presented in General Rebuttal:

• Existence: The average token duplication rate is up to 99% on all math tasks, and even a staggering 99.9% on some simple arithmetic tasks; In contrast, the token duplication rate on the text generation task is only about 60%, with about 2000 different types of token, and still increasing as the total number of tokens increases. These data and comparisons demonstrate that pattern collapse occurs in mathematical reasoning and not in text generation.

• Universality: Token repetition rate exceeded 97% on all seven math tasks of different difficulties and types. This demonstrates the universality of "pattern collapse" in various mathematical reasoning tasks.

W2: I am not convinced that early stabilization is generally held for mathematical reasoning problems. More evidence should be provided to justify it.

1. Setup Description of Figure 3

The ID data curve is the average of all samples in the MultiArith dataset, and the OOD data curve is the average of all samples in the five domains of MATH dataset (Algebra, Geometry, Counting and Probability, Number Theory, and Precalculus). These are consistent with our settings for the ID and OOD datasets in the experimental setup (Section 4.1). We will add this setup in the updated version.

2. More evidences about "early stabilization is generally held for mathematical reasoning problems"

Please refer to "General Rebuttal: Universality of "Early Stabilization" phenomenon in mathematical reasoning (Expanded visualization of Figure 3 in the paper)" for detailed responses and evidences.

W3: The writing is also not precise. ...the domain of f is not specified. The notation \phi is not defined.

Thanks for pointing out some questions about paper writing.

• The domain of $f(\boldsymbol{x}, \boldsymbol{y}, \boldsymbol{\theta})$ is any value $(\boldsymbol{x}, \boldsymbol{y}, \boldsymbol{\theta})$ sampled from the joint probability distribution $P_{\mathcal{X} \times \mathcal{Y} \times \Theta}$ defined on the $\mathcal{X} \times \mathcal{Y} \times \Theta$, where $\mathcal{X}, \mathcal{Y}, \Theta$ are the input space, the output space, and the parameter space, respectively, as defined in lines 70-75. We did not specify this because there are no special values that need to be excluded.

• The notation "\phi" does not appear in our paper.

We will improve our presentation in updated version.

W4: The experiment is not solid. There is no experiment to justify that the performance improvement is due to addressing the challenges posed in the Introduction.

1. Response to "There is no experiment to justify that performance improvement is due to addressing the challenges ..."

• First, the two "challenges" are two objectively existing phenomena in mathematical reasoning scenarios. However, they make existing embedding-based methods that cannot be applied to mathematical reasoning, so we call them "challenges" for existing methods, not for us. As stated in lines 38-39, “However, embedding-based methods encounter challenges under mathematical reasoning scenario”. These two challenges are simply introduced to explain why existing methods cannot be applied to mathematical reasoning;

• Second, because existing methods are hindered by these two phenomena, we need to find new methods to circumvent encountering them. Thus, we aim to circumvent the two challenges faced by existing methods, not to address them. As stated in lines 47-48, “Therefore, we transform our perspective from the static embedding representation to the dynamic embedding shift trajectory in the latent space", we are not improving on existing methods, but rather two different research ideas.

• Third, these two phenomena are challenges for existing methods, but opportunities for us: we utilize them to design trajectory-based methods (Section 2-3).

2. Reasons for performance improvement

A complete explanation of why our method yields performance improvement is given in Section 2 (motivation part).

Refer to "General Rebuttal: Motivation line from "pattern collapse" to "early stabilization" and TV score" for details.

3. Response to “The experiment is not solid”

We have performed rich dataset experiments, scalable experiments, ablation analyses, and failure analyses in Sections 4-5 and Appendices E-G to demonstrate the solidity of our experiments.

We expect these responses to address your concerns, and look forward to your more positive comments.

Thanks for your constructive comments. We will respond to your concerns one by one.

W1: The input data for the empirical study in Figure 3 is not introduced. Thus, whether it can reflect the scenario of mathematical reasoning is not clear.

Thanks for pointing this out, and sorry for the missing experimental setup (i.e., input data source) for Figure 3.

1. Setup Description of Figure 3

The ID data curve is the average of all samples in the MultiArith dataset, and the OOD data curve is the average of all samples in the five domains of the MATH dataset (Algebra, Geometry, Counting and Probability, Number Theory, and Precalculus). These are consistent with our settings for the ID and OOD datasets in the experimental setup (Section 4.1). We will add this setup in the updated version.

2. More evidences about "it can reflect the scenario of mathematical reasoning"

Please refer to "General Rebuttal: Universality of "Early Stabilization" phenomenon in mathematical reasoning (Expanded visualization of Figure 3 in the paper)" for detailed responses and evidences.

W2 & Q1: The relationship of the pattern collapse (in Figure 1) and the early stabilization (in Figure 3) is not clearly illustrated.

Please refer to "General Rebuttal: Motivation line from "pattern collapse" to "early stabilization" and TV score" for detailed responses.

Thanks for pointing out the vague elaboration on the motivation line. The pattern collapse (in Figure 1) and the early stabilization (in Figure 3) are not directly related, but indirectly transitioned through the theoretical intuition of Section 2.1. To summarize, the "pattern collapse" leads to more significant trajectory differences across different samples, and the source of the trajectory differences between ID and OOD samples is the "Early Stabilization" phenomenon.

In detail, the function of each section is:

• In Section 1, we find the "pattern collapse" (in Figure 1) in the output space;
• In Section 2.1, we introduce the intuition: the presence of "pattern collapse" causes the convergence of the trajectory endpoints of different samples, leading to significant trajectory differences across samples (As stated in Lines 104 Hypothesis 1). We justify this intuition through theoretical modeling and proving.
• We already have the intuition that trajectories can be a good measure, but it is still unknown what kind of difference exists between the trajectories of the ID and OOD samples. Thus, in Section 2.2, we conducted empirical experiments to observe the trajectory volatilities of different scenarios and discover the phenomenon of "early stabilization" (in Figure 3), which is the root cause of the trajectory differences.

We will state this motivation line at the start of Section 2 in the updated version.

W3: The datasets for experiments are quite limited.

Thanks for pointing out the limited datasets. We illustrate this in terms of both data type and data size:

1. Data Type

As for the data type, we have collected 10 of the most commonly used datasets in the LLM mathematical reasoning research for our experiments, and they cover six types of mathematical tasks and all levels of difficulties from elementary school to college. Due to space limits, we reported the average results for each setting in the main text, and the results on each dataset have been shown in Appendix E (Tables 8-11). Overall, our dataset types have been as rich as possible.

2. Data Size

Data size is a limitation as we mentioned in Section Limitation. However, this is caused by the particular nature of the mathematical reasoning research field, and is not a subjective constraint on us, for the following reasons:

• Compared to traditional text generation tasks such as summarization and translation, mathematical reasoning tasks did not receive much attention prior to the era of LLMs, and thus there are few general-purpose datasets. At the same time, automated construction of mathematical datasets needs to be based on complex rules, resulting in complex mathematical tasks requiring significant and time-consuming human intervention[1,2], making diversity and size severely limited.
• After the advent of the Chain-of-Thought technique[3], mathematical reasoning started to receive more attention from the NLP field, but some of the more complex mathematical tasks are usually built for LLM evaluation[4], such as TheoremQA (600 samples)[5], SAT-Math (220 samples)[6], MMLU-Math (974 samples)[7], making the large-scale training sets not specifically constructed. All these factors make the dataset size in the field of mathematical reasoning much smaller than traditional generative tasks.

On the other hand, we are the first to study OOD detection on mathematical reasoning, and the current data type is sufficient to demonstrate the generality of our method. If new large-scale mathematical reasoning datasets become available, our method can be further applied.

[1] Solving general arithmetic word problems, EMNLP 2015.

[2] Measuring mathematical problem solving with the math dataset, NeurIPS 2021.

[3] Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, NeurIPS 2022.

[4] MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning, ICLR 2024.

[5] TheoremQA: A Theorem-driven Question Answering Dataset. ACL 2023.

[6] Agieval: A human-centric benchmark for evaluating foundation models. NAACL 2024.

[7] Measuring massive multitask language understanding. ICLR 2021.

We expect the above responses to address your concerns, and look forward to your more positive comments.

Thanks for your constructive comments! We will respond to your concerns one by one.

W1 & Q1: The mechanism of TV score ... more analysis and visualization of trajectory ... intuition behind the choice of trajectory volatility ...

1. Mechanism of TV score and its relationship to "pattern collapse"

The relationship of TV score and "pattern collapse" are not directly related, but indirectly transitioned through the theoretical intuition of Section 2.1. Refer to "General Rebuttal: Motivation line from "pattern collapse" to "early stabilization" and TV score" for details.

2. Intuition behind the choice of trajectory volatility

The Intuition is our Hypothesis 1 (line 104): The "pattern collapse" in mathematical reasoning scenarios leads to more significant differences among different samples' trajectories compared to traditional generative tasks (we have modeled and proved this intuition from a theoretical perspective in Section 2.1)

3. Detailed analysis and visualization of trajectory dynamics

Refer to "General Rebuttal: Universality of "Early Stabilization" phenomenon in mathematical reasoning (Expanded visualization of Figure 3 in the paper)" for detailed responses.

W2 & Q2: The computational complexity is not discussed ...

Thanks for your consideration. After obtaining all outputs $\boldsymbol{y_l}$ of each layer, there are two main steps to obtain the final scores:

(1) Get ID Data Information: Fitting Gaussian distribution $\mathcal{G}_l$ = $\mathcal{N}(\boldsymbol{\mu}_l, {\boldsymbol \Sigma}_l)$ for $L$ layers

${\boldsymbol \Sigma}_l$ is a diagonal matrix because $d$ embedding dimensions are independent, so we only need to compute the mean and variance of each dimension of all $n$ ID sample embeddings.

• Compute mean ${\boldsymbol{\mu}}_l$: $d(n-1)$ addition operations and $d$ multiplication operations are required;
• Compute variance ${\boldsymbol \Sigma}_l$: $dn+d(n-1)=d(2n-1)$ addition operations and $dn+d=d(n+1)$ multiplication operations are required.

We fit $L$ Gaussian distributions, so the number of addition and multiplication operations are both $\mathcal{O}(Ldn)$.

We report the computation time of fitting Gaussian distribution in the ID dataset MultiArith ($n$=600):

Note: this part is a one-time event and only needs to be performed once.

(2) Get OOD Data Information: Compute TV Score

For $k=0$, we need to compute the Mahalanobis Distance between OOD sample and ID distribution (Eq 6). Since ${\boldsymbol \Sigma}_l$ is a diagonal matrix, it involves only simple vector multiplication and does not require matrix multiplication. Specifically, it requires $d + (d-1) = 2d-1$ addition operations and $2d$ multiplication operations. We have $L$ layers, so the numbers of addition and multiplication operations are both $\mathcal{O}(Ld)$.

For $k>0$, for each increase in $k$ by 1, $L$ Gaussian distribution differences, $L$ embedding differences, and $L$ MD computations need to be performed. The increasing numbers of addition and multiplication are both $\mathcal{O}(Ld)$.

We sample 1000 cases to compute TV score. Time is as below:

After complexity analysis and experimental results, it is clear that our method is efficient and can be flexibly deployed in realistic scenarios. This is one of our strengths, especially compared to some probability-based metrics such as perplexity, which require time-consuming softmax exponential computation.

W3 & Q3: The paper does not address the robustness ...

Thanks for your consideration. However, prior works on OOD detection did not take this into account, so the robustness of OOD detection is not a well-defined question. Despite this, we try our best to design experiments to verify this point.

We make the following assumption and goal: For a perturbed ID sample, the sample still belongs to the ID sample in realistic scenarios, but models may misidentify it as an OOD sample. We need to avoid this misidentification.

We refer to [1] for perturbation method to input data in language models:

• Paraphrasing: We generate one paraphrased input by querying ChatGPT using the prompt in [1];
• Dummy Tokens: We randomly select tokens that marginally influence the original meaning and append them to the input. Such tokens could be newline characters, tab spaces, ellipses, or supplementary punctuation marks.

We compare two settings:

• Original: Results in our paper
• Perturbation: We replace ID samples in the test set with the perturbed text. We report results under two perturbations.

Our method can largely defend against some perturbations in realistic scenarios, showing its strong robustness. We conjecture that this is because our method considers a large amount of information in the middle layer, whereas probability- or embedding-based methods only consider a single layer in the output/input, which makes our method better resistant to the influence of some random factors in the output layer, such as overconfidence, and therefore more robust.

As for more explorations, we leave it for future work.

[1] SPUQ: Perturbation-Based Uncertainty Quantification for Large Language Models. EACL, 2024.

We expect these responses to address your concerns, and look forward to your more positive comments.

Dear Reviewer SAjz,

We want to express our sincere appreciation for your efforts and time spent reviewing our work and the constructive comments.

During the rebuttal period, we have provided a detailed response to address all the concerns mentioned by you about mechanism, complexity, and robustness. With the reviewer-author discussion period ending in one day, we kindly invite you to give some feedback on our responses, and we really hope our responses adequately address your concerns.

Best,

Authors