W3: A more detailed explanation on how to identify the correctness

Happy to address your question, and I pinpoint it to line 320. In fact, we have aligned with the evaluation framework that OpenAI used when evaluating the GPT series models (see Appendix C.2.3, line 1442, https://github.com/openai/simple-evals). I will use the MGSM dataset as an example to explain how to identify the correctness of LLM responses (i.e., how to extract the ground-truth correctness labels).

Due to what we used being instruct-based models, we prompt LLMs to follow specific instruction formats when generating answers, which facilitates our answer extraction. For example:

Solve this math problem. Give the reasoning steps before giving the final answer on the last line by itself in the format of "Answer:". Do not add anything other than the integer answer after "Answer:".
Question: {input_data}

When LLMs receive the above instructions and questions, it will follow the format to generate the following contents:

[......]
Answer: [...]

Now, we only need to use regular expression matching to extract the content that follows "Answer: " to obtain the exact answer generated by the LLM. If it matches the true label, the correctness label is set to 1; otherwise, it is set to 0. All instructions have been presented in Appendix C.2.3.

---

W4 & Q4: Experiments --- Layer number analysis

I agree with your viewpoint that since there is a correlation between the CoE and the layer number, it is meaningful to analyze the impact of the layer number on performance.

In fact, among the seven LLMs we selected, the influence of the layer numbers on performances can already be concluded in Table 1 due to the differentiation in parameter scales. We label the layer numbers of these LLMs and copy the AUROC results of the CoE-R metric as follows:

• The first five LLMs are all 7B+ models with 28/32 layers; the last two LLMs are both 70B+ models with 80 layers:

  • From the results, on 3/4 of the domains, LLMs with more layers (80 layers) significantly outperform those with fewer layers (30 layers). Although we cannot definitively conclude that "more layers always mean better performance," there is indeed a noticeable trend.
  • This trend is quite reasonable: as the number of layers increases, the amount of information contained within the model grows, allowing for more features in the trajectory to effectively distinguish the correct samples.
  • Based on this analysis, our method shows promise: As the demand for larger-scale LLMs (more parameters and more layers) surges in the industry, the enhanced model scaling robustness allows our method for widespread deployment in real-world scenarios, ensuring its broad generalizability.

---

Additionally, this work serves as a foundation that can be extended to broader applications, such as decoding and preference optimization (Refer to our discussion with Reviewer 2nBf).

Finally, thank you once again for taking the time to review our work and provide valuable insights. We hope our response can address your concerns and that you can recognize the value of our work. We look forward to your more positive feedback.

Thanks for your constructive feedback. We respond to your questions one by one:

---

W1 & Q1 & Q2: Scenarios and the necessity for the label-free scenario ... If open-world scenarios: (1) How to handle distributional differences in a single domain? (2) How to distinguish trajectories across diverse domains?

Thank you for your deep reflections on the task setting. This is an open-ended question, as previous work in the field of self-evaluation has been conducted on identically distributed data --- obtaining classifiers on identically distributed data is a common practice. The open-world assumption you mentioned has not appeared in prior research, but we are willing to explore this further to validate the generalizability of our work.

Q1: How does the proposed method handle distributional differences across open-world queries?

We consider the scenario of different distributions within the same domain that you mentioned. We mix GSM8K and MATH datasets (Mathematics Domain) / CommonsenseQA and TheoremQA (Reasoning Domain) used in our paper, as they have significantly different data sources and problem difficulties, to simulate this scenario. To ensure data balance, we set the number of samples in the two datasets consistent.

We report AUROC results upon Llama3-8B-Instruct and Qwen2-7B-Instruct models as follows:

a. Mathematics (GSM8K + MATH)

b. Reasoning (CommonsenseQA + TheoremQA)

It is clear that our method is fully capable of handling scenarios with different distributions within the same domain, and it significantly outperforms other baseline methods. This validates the generalization of our method under the open-world assumption.

Q2: Can the method reliably distinguish trajectories across diverse domains?

Of course, since the trajectory feature differences between different domains are significant, we can cluster any new samples in an open world based on the acquired domain priors. We have verified that the data distributions within the same domain show minimal differences (Q1), so we can obtain the domain prior CoE scores using a small amount of data. After that, we determine which domain's CoE prior score is closest to the CoE score of any new sample to classify its belonging domain. Based on this method, we validate the clustering accuracy:

We find that all accuracies are higher than 95%, indicating that our method can effectively distinguish trajectories from different domains.

W2 & Q3: There is no evidence to indicate the correspondence between the human thinking path and LLM intermediate path. Moreover, there may many knowledge-intensive task may not need the thinking path.

Thank you for pointing this out, we will clarify our motivations as follows:

What is the evidence for aligning human thinking paths with LLM intermediate paths?

We do not need to provide evidence for the alignment of human and LLM thinking paths, as all descriptions of human thinking paths are heuristic for our LLM research. The logic of our motivation is as follows:

● Firstly, we claim that there are differences in the thinking path of the human brain when considering right and wrong. We want to draw an analogy to assume that "LLM thinking paths also exhibit differences". Our focus in this analogy is on the behavior of "exhibiting differences" rather than on the concept of "thinking path". (The fundamental logic here is: This analogy assumption does not need to be based on the alignment of the thinking paths of humans and LLMs.)

● Based on this analogy assumption, we only need to define the thinking path of the LLM and verify this assumption. As stated in the introduction, the LLM models syntax at a low level and semantics at a high level, which constitutes a latent thinking path.

Overall, our motivation is heuristic, focusing on the "differences in thinking paths." The core of our paper is to validate the differences in thinking paths in LLMs. This does not need to be grounded in the alignment between LLMs and humans, nor does it require validating the compatibility between the two through the conclusions of the paper. In fact, Both Reviewer 2nBf and GkSP recognize the rationality of our motivation and affirm this strength.

Moreover, there may many knowledge-intensive task may not need the thinking path.

I believe you have confused our concept of "latent thinking paths" with "Chain-of-Thought (CoT)", which is a post-hoc thinking process that is reflected in the model's output rather than within the model itself. From the view of LLM-generated answers, responding to such questions indeed may not require specific steps, but this feature claimed by you points to the concept of CoT.

As for latent thinking paths, even for knowledge-based tasks, there will certainly be steps involving modeling syntax, grammar, semantic information, and memory retrieval. Although it may not be clearly explainable, these latent processing definitely exists.

W4: Experiments --- Robustness analysis on adversarial samples

Thank you for your consideration. To the best of our knowledge, the previous self-evaluation work did not take into account adversarial robustness, which is not a well-defined problem in this research area. Upon reflection, we realize that perturbing the samples might affect the accuracy of the LLM's responses, which would change the ratio of positive to negative samples. Based on this, we can even treat the evaluation of the perturbed data as a completely new evaluation setting on a separate set of independent data.

Of course, we fully support your concerns for it is important for real-world employment. Therefore, we refer to [1] to construct two batches of perturbation data, with the following construction methods:

• Paraphrasing: We generate one paraphrased input by querying ChatGPT using the prompt in [1];
• Dummy Tokens: We randomly select tokens that marginally influence the original meaning and append them to the input. Such tokens could be newline characters, tab spaces, ellipses, or supplementary punctuation marks.

We compare two settings:

• Original: Results in our paper
• Perturbation: We replace raw samples in the dataset with the perturbed data. We report results under two perturbations.

We report AUROC results upon Llama3-8B-Instruct and Qwen2-7B-Instruct models as follows:

---

(a) Llama3-8B-Instruct

Domain I: Mathematics

Domain II: Reasoning

Domain III: Knowledge

Domain IV: Understanding

---

(b) Qwen2-7B-Instruct

Domain I: Mathematics

Domain II: Reasoning

Domain III: Knowledge

Domain IV: Understanding

---

After perturbing the raw data, the performance of our method remains stable, and is better than other uncertainty estimation baselines. We find that compared with our method, some uncertainty estimation methods, such as PPL, exhibit an overconfidence phenomenon in LLMs. When faced with perturbations, LLMs may produce incorrect answers for samples that were originally answered correctly, yet their output probabilities remain relatively high.

These results and conclusions indicate that our method exhibits sufficient robustness against adversarial perturbations.

[1] SPUQ: Perturbation-Based Uncertainty Quantification for Large Language Models. EACL, 2024.

Q1: Is there a threshold to decide the classification using metrics?

Yes, we can use the AUROC curve to obtain a threshold, this is a standard way as prior work. We refer to [1]: "We computer the optimal cut-off $\tau_i$ of Youden Index, which is at the point in the AUROC curve where TPR-FPR is maximum." (The Youden Index is a standard statistical measure used to evaluate the performance of classifiers.)

After obtaining the threshold $\tau_i$, for each sample, if the metric score (CoE or other baselines) is greater than $\tau_i$, it is classified as correct; otherwise, it is classified as incorrect. Based on this criterion, we can derive a threshold after obtaining an AUROC curve for each dataset/model/metric, and then calculate the Accuracy.

We report Accuracy results upon Llama3-8B-Instruct and Qwen2-7B-Instruct models as follows:

Domain I: Mathematics

Domain II: Reasoning

Domain III: Knowledge

Domain IV: Understanding

After obtaining the threshold and calculating the classification accuracy, our method still maintains optimal performance.

[1] Embedding Trajectory for Out-of-Distribution Detection in Mathematical Reasoning. NeurIPS 2024.

---

Q2: Why does the average embedding at layer l represent the l-th sentence hidden state?

Regarding the representation of the sentence hidden state, we strictly align with the definitions in the previous papers, as shown below:

• [1] ... To obtain the output embedding, we average the decoder’s final-layer hidden state vectors.
• [2] ... the sentence embedding can be obtained by averaging the token embedding ...
• [3] ... we define the average embedding as the sentence embedding at layer l.

A reasonable explanation is that each token in a sentence contributes to the overall semantics of the sentence. In fact, in traditional sentence embedding research, this approach has already existed as a naive modeling way[4,5].

[1] Out-of-Distribution Detection and Selective Generation for Conditional Language Models. ICLR 2023.

[2] INSIDE: LLMs' Internal States Retain the Power of Hallucination Detection. ICLR 2024.

[3] Embedding Trajectory for Out-of-Distribution Detection in Mathematical Reasoning. NeurIPS 2024.

[4] SimCSE: Simple Contrastive Learning of Sentence Embeddings. EMNLP 2021.

[5] Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. EMNLP 2019.

---

Additionally, this work serves as a foundation that can be extended to broader applications, such as decoding and preference optimization (Refer to our discussion with Reviewer 2nBf).

Finally, thank you once again for taking the time to review our work and provide valuable insights. We hope our response can address your concerns and that you can recognize the value of our work. We look forward to your more positive feedback.

W1: Missing critical statistic analysis of discrepancies.

Thank you for pointing this out. In Section 2, Figure 2 has already represented the quantitative data analysis you mentioned, while Figure 3 is the qualitative visualization analysis. In Figure 2, we present the statistical data (i.e., the Magnitude and Angle values of all samples) in the form of a two-dimensional distribution, only allowing readers to intuitively grasp the patterns behind the data.

Of course, we appreciate your rigorous considerations, so we report all data points from Figure 2 (including the mean and standard deviation of each feature) as follows:

Feature 1: Magnitude

Feature 2: Angle

The statistics are consistent with the patterns presented in Figure 2. I hope our clarification can help you eliminate any misunderstanding you have regarding this weakness.

W3: Experiments --- triviaQA and TruthfulQA datasets

Thanks for your mention. We report AUROC results on the two datasets upon Llama3-8B-Instruct and Qwen2-7B-Instruct models.

(a) triviaQA

(b) TruthfulQA

From the results, our method still demonstrates excellent performance on these two datasets.

Thanks for your constructive feedback. We respond to your questions one by one:

---

W2: Why is the targeted task self-evaluation? self-evaluation is a task that seems appealing but is quite ambiguous and questionable; even if the reviewer reads the reference paper cited by authors for self-evaluation, there is no task definition of self-evaluation... (1) Self-evaluation is highly associated with uncertainty estimation... (2) Self-evaluation is associated with llm-as-a-judge and self-rewarding...

Thank you for your thorough reading and deep reflection on our paper. Regarding the self-evaluation setting, this is a point worth discussing, and we organize our views as follows:

The definition of self-evaluation is ambiguous and questionable

The definition of self-evaluation is a relatively ambiguous concept, and previous papers indeed do not completely give its definition. First, from [1], we can identify the nascent concept of self-evaluation [2], whose research objective is to "explore whether LLMs are aware of the correctness of their responses." Considering the ambiguity of the definition, we can only integrate the method categories that align with self-evaluation research objectives, thereby forming a methodological framework for self-evaluation:

• The related work in [3] comprehensively delineates the scope of methods involved in self-evaluation, including Verbal Confidence and the sampling-based Prompt-Sampling-Aggregation (PSA) Pipeline.
• The introduction in [4] also clearly categorizes methods based on logits/probabilities as a category under self-evaluation, which actually aligns closely with the traditional scope of uncertainty estimation that you mentioned. Based on these backgrounds, we integrate a framework of methods that aligns with the research objectives of self-evaluation, and categorize according to characteristics:

Based on these backgrounds, we integrate a framework of methods that aligns with the research objectives of self-evaluation, and categorize according to characteristics:

Note that these three categories are aligned with our introduction (paraphrase 2) and the experimental baselines.

Why is the targeted task self-evaluation?

Our work aligns with the research objectives of self-evaluation, but it does not belong to any of the three method categories mentioned above. Therefore, we tend to define our study under the concept of self-evaluation and discuss and compare it with other method categories that have the same research objective.

Self-evaluation is highly associated with uncertainty estimation. The authors don't mention their targeted method for uncertainty estimation.

As we integrated above, our method and uncertainty estimation methods are parallel, both serving the same research objective of measuring the likelihood of "LLM answering correctly" through a scalar score. However, our method falls into the fourth category:

Indeed, our CoE method is closest to the uncertainty estimation for we are both white-box methods, but we cannot categorize our method within the realm of uncertainty estimation for we access different components inside the LLMs.

Self-evaluation is associated with llm-as-a-judge and self-rewarding. ... For label-free self-evaluation, I advise authors to compare their metrics with llm-as-a-judge or self-rewarding methods (label-based self-evaluation) in some instruction-following tasks, even not sota.

Thanks a lot for your kind suggestion, but I’m sorry that self-evaluation is completely contradictory with LLM-as-a-judge and label-based evaluation.

To be simplest, we just need to go back to the self-evaluation research objective[2] of "explore whether LLMs are aware of the correctness of their responses." — this means that our research focus is on the LLMs themselves. For LLM-as-a-judge, they investigate whether an external LLM can be used to judge the LLM of our study, so their research focus is on external LLMs; and for label-based evaluation, it is completely unnecessary to consider the LLM's own awareness of what it knows. Therefore, self-evaluation and the two types of research you mentioned are not on the same track.

Dear Reviewer GkSP,

We first thank you again for your constructive comments. We have addressed your concerns one by one and supplemented more detailed experimental results and explanations. We look forward to further discussion with you and your positive feedback about our rebuttal.

Best regards,

Authors

how can these findings be used to improve the reasoning accuracy of LLMs during pre-training or post-training?

First, thanks very much for your recognition of our work. Regarding your only question, we would also be happy to discuss the future application value of the CoE with you. From our view, we think that these findings can be used to improve the reasoning accuracy of LLMs from the following topics:

(1) Inference-time Decoding

In this work, we find that CoE can serve as a measure of the latent thinking paths of LLMs and can distinguish between correct and incorrect samples. This allows us to apply it to decoding sampling during inference time: similar to self-consistency[1], but it only considers voting on the exact answers to the output text. CoE, on the other hand, brings in more potential information from within the model, enabling us to sample multiple trajectories and select the most suitable one as the final voting result.

(2) Preference Optimization

Since we can use CoE to obtain a score (CoE-R and CoE-C) that reflects the correctness of the LLM responses, we can use it as a scorer for preference data pairs, thereby enabling self-iterative preference optimization in the post-training phase. This idea of self-iterative PO is similar to a recent work[2], but it demonstrates a significantly higher efficiency using single trajectories rather than multiple sampling.

In general, this paper can serve as an inspirational study for interpretable mechanisms that have many application values in the future. There must be more valuable scenarios waiting to be explored. We welcome your suggestions and more positive feedback.

[1] Self-Consistency Improves Chain of Thought Reasoning in Language Models. ICLR 2023.

[2] Self-Consistency Preference Optimization.