How Post-Training Reshapes LLMs: A Mechanistic View on Knowledge, Truthfulness, Refusal, and Confidence

Question #4: Did you explore other important axes such as reasoning, robustness, or chain-of-thought abilities? Why were these four perspectives chosen?

We selected these four perspectives because we think that they represent fundamental aspects of language model behaviors, and they can be studied through well-defined experimental designs using available interpretability tools. While we acknowledge that other axes such as reasoning, robustness, and chain-of-thought abilities are also very important for understanding model behaviors and improving model performance, these areas present significant methodological challenges for existing interpretability tools. Specifically, reasoning and CoT abilities involve complex, multi-step processes that are difficult to isolate and measure in controlled, curated settings. We can analyze the model's output and performance as the reasoning ability, but it is difficult to analyze its internal mechanism. We view the development of more sophisticated interpretability tools capable of dissecting complex cognitive processes like reasoning as a promising direction for future research. We are also eager to find out how post-training reshapes the reasoning ability of LLMs.

Reasons to reject: While the results are empirical and insightful, some deeper theoretical grounding is missing. The confidence section feels less robust compared to others; it largely concludes ``entropy neurons aren’t it’’ without exploring other possible factors.

Since LLMs are too complex, theoretical study is very difficult in this field. Nonetheless, there have been many useful interpretability works that give insights but not theory. The empirical results already provide good intuitions and could already benefit many applications, “could potentially lead to developing innovative ideas in this space”, as mentioned by Reviewer bJdd. For example, the intuitive understanding of the linear directions could benefit transfer-intervention by the linear direction of the Base model to the post-trained model or vice versa. We would be willing to extend our paper in the future if new theories could concretely explain these phenomena. For your concern about the entropy neurons, we have answered it in the questions above.

Thank you again for your feedback!

Reference:

[1] Samuel Marks and Max Tegmark. 2024. The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets.

[2] Lennart Burger, Fred A. Hamprecht, and Boaz Nadler. 2024. Truth is universal: Robust detection of lies in llms.

[3] Andy Zou, Zifan Wang, J. Zico Kolter, and Matt Fredrikson. 2023. Universal and transferable adversarial attacks on aligned language models.

[4] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model.

[5] Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. 2024. Refusal in language models is mediated by a single direction.

[6] Alessandro Stolfo, Ben Wu, Wes Gurnee, Yonatan Belinkov, Xingyi Song, Mrinmaya Sachan, and Neel Nanda. 2024. Confidence regulation neurons in language models.

[7] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model.

[8] Kaitlyn Zhou, Jena D. Hwang, Xiang Ren, Maarten Sap. 2024. Relying on the Unreliable: The Impact of Language Models’ Reluctance to Express Uncertainty.

Thank you for recognizing our impact, choosing four inspiring perspectives, and the practical importance! We would like to address your questions and concerns respectively as follows.

Question #1: Why is backward transfer (POST to BASE) possible for refusal but not forward (from BASE to POST)? Any hypotheses?

We hypothesize that maybe the POST models develop a more sophisticated refusing mechanism with a more specified refusal direction. The acceptable refusal direction range in the POST model might be narrower than the BASE model, so the refusal direction in the BASE model does not work well in the POST models, but the refusal direction of the POST models might work well in the BASE model. It could be an interesting future direction to verify this hypothesis and dig deeper into the refusal behavior. This detailed analysis itself could become a separate paper. Our paper aims to provide a systematic analysis of the four perspectives that we think are important.

Question #2: Beyond entropy neurons, did you investigate other hidden state characteristics that might explain confidence differences?

During our initial exploration, we considered developing an output-centered approach, where we rely on the model itself to express its confidence and try to attribute this confidence to model internal modules. However, we later found that letting the model itself determine if its output is confident or not is not reliable. As also pointed out by previous work [8], language models are reluctant to express uncertainties when answering questions even when they produce incorrect responses, and when explicitly prompted to express confidence, they tend to be overconfident. This is why we eventually chose the entropy neuron as our interpretability tool as it is intrinsic to the model.

We acknowledge that the confidence section has some limitations. However, we think this finding is still valuable to report. Confidence calibration is an important part of understanding post-training's effects, and we identified entropy neurons as a potentially suitable tool to study this perspective based on prior works. Even though entropy neurons turned out not to be useful for this task, we believe reporting these results contributes valuable knowledge about what approaches don't work in this domain. It remains a future work to explore what leads to confidence change during post-training.

Question #3: What models (sizes, architectures) and datasets were specifically used?

As explained in Section 3, we mainly conduct experiments on two representative LLM model families: Llama-3.1-8B and Mistral-7B-v0.3. For each of these two model families, we conduct experiments on the base model, instruction-tuned model, and SFT model. We use the base model and instruction-tuned model from the officially released model, and we use the SFT models from Llama-3.1-Tulu-3-8B-SFT and Mistral-7B-Base-SFT-Tulu2, which were finetuned on the Tulu dataset, since there was no official released SFT models of this size.

For datasets, we choose the appropriate dataset for each analyzing perspective following previous papers, so that we can make sure our setting is reasonable and appropriate. For the knowledge and truthfulness perspectives, we use datasets from [1], where each dataset contains simple and unambiguous statements that are either true or false from diverse topics. For example, the cities dataset contains statements about cities and their countries, following the format “The city of [city] is in [country]”. To eliminate the concern that the datasets might be out-of-distribution for post-training, we curate a dataset that is in-distribution for SFT models. We curate the Tulu extracted dataset from the tulu-3-sft-mixture dataset that was used to finetune the Llama-3.1-8B-SFT model. We ensure every statement from the extracted dataset also appears in the tulu-v2-sft-mixture dataset that was used to finetune the Mistral-SFT model. For experiments on the refusal perspective, we follow [5] to use advbench [3] for harmful inputs and alpaca [7] for harmless inputs. The confidence perspective directly analyzes the entropy neurons, so we do not need any dataset. Details about our datasets are shown in Appendix A.

Do you expect findings to generalize to much larger or more instruction-heavy models (e.g., GPT-4)?

To verify whether our findings generalize to larger models, we conduct experiments on Llama-2-13B and Llama-2-13B-Instruct models. All our previous conclusions are consistently verified on this 13B model.

For the knowledge storage perspective, we use the same experiment settings as our paper and use the metric as Table 1 to evaluate the causal tracing results. We conduct experiments on the cities, sp_en_trans, and tulu_extracted datasets. The results are shown below:

The causal tracing results of the based model and instruction-tuned model have a high correlation coefficient, and their maximum difference is low. It verifies our previous conclusion that post-training has little influence on knowledge storage locations.

For the truthfulness probing experiments, we follow the same setting and metrics as Table 2, and the results are shown below:

For the truthfulness intervention experiments, we follow the same setting and metrics as Table 3, and the results are shown below:

For the refusal intervention experiments, we follow the same setting and metrics as Table 4, and the results are shown below:

These results confirm our previous conclusions that truthfulness directions are similar between the BASE and POST models but the refusal directions are different.

For the entropy neuron experiments, all 10 top entropy neuron candidates are the same between the base model and instruction-tuned model. We also calculate the ratio of weight norm divided by the log LogitVar for the identified entropy neurons (as in Table 16 in Appendix E). The average ratio difference is only 0.003075 between the base model and the instruction-tuned model. This confirms our conclusion that the confidence difference between BASE and POST models cannot be attributed to entropy neurons.

Unfortunately, we do not have the resources to conduct experiments on even larger models, but we expect our findings to generalize, and it remains a future work to verify them.

Thank you very much for your positive feedback and recognition of our findings and potential benefits to real-world applications!

Evaluating on larger models, such as those with 13 billion or even 40 billion parameters, could yield better results. Do the larger models behave consistently across these four perspectives?

To verify whether our findings generalize to larger models, we conduct experiments on Llama-2-13B and Llama-2-13B-Instruct models. All our previous conclusions are consistently verified on this 13B model.

For the knowledge storage perspective, we use the same experiment settings as our paper and use the metric as Table 1 to evaluate the causal tracing results. We conduct experiments on the cities, sp_en_trans, and tulu_extracted datasets. The results are shown below:

The causal tracing results of the based model and instruction-tuned model have a high correlation coefficient, and their maximum difference is low. It verifies our previous conclusion that post-training has little influence on knowledge storage locations.

For the truthfulness probing experiments, we follow the same setting and metrics as Table 2, and the results are shown below:

For the truthfulness intervention experiments, we follow the same setting and metrics as Table 3, and the results are shown below:

For the refusal intervention experiments, we follow the same setting and metrics as Table 4, and the results are shown below:

These results confirm our previous conclusions that truthfulness directions are similar between the BASE and POST models but the refusal directions are different.

For the entropy neuron experiments, all 10 top entropy neuron candidates are the same between the base model and instruction-tuned model. We also calculate the ratio of weight norm divided by the log LogitVar for the identified entropy neurons (as in Table 16 in Appendix E). The average ratio difference is only 0.003075 between the base model and the instruction-tuned model. This confirms our conclusion that the confidence difference between BASE and POST models cannot be attributed to entropy neurons.

Unfortunately, we do not have the resources to conduct experiments on even larger models, but we expect our findings to generalize, and it remains a future work to verify them.

Thank you for recognizing our paper as timely and impactful, thorough and generalizable, and mechanistically insightful!

Section 7 is rather not well-written. While explained briefly, the paper could use explaining how entropy neurons were applied in their analysis in a bit more detail.

We agree that the writing of Section 7 can be further improved. The current discussion of entropy neurons is insufficient, mainly due to space limitations. In our updated version, we will include the following discussion about our methodology of applying entropy neurons:

Entropy neurons are identified by considering weight norm and logit attribution. First, we compute the logit attribution for each neuron in the final MLP layer by projecting their output weights onto the vocabulary space through the unembedding matrix, as shown in Equation 4 of our paper. This projection approximates each neuron's direct effect on the final prediction logits (its effect on each token's output logit). We then calculate the variance of this normalized projection (LogitVar), where low LogitVar indicates a relatively balanced contribution across all vocabulary tokens rather than promoting specific tokens. Entropy neurons typically have a large weight norm (so that they are influential) but low LogitVar (so that they have a relatively balanced contribution across all vocabulary tokens). Our identification process first selects the top 25% of neurons with the largest weight norms, and from this subset, we identify the 10 neurons with the lowest LogitVar values as entropy neurons. This methodology follows established practices from prior work [1,2] and captures neurons that modulate output entropy without significantly affecting token ranking.

In our analysis comparing BASE and POST models, we found substantial overlap in identified entropy neurons, with a highly similar ratio of weight norm divided by log LogitVar (Table 16 in Appendix E). It suggests that the confidence regulation mechanism of entropy neurons remains largely unchanged during post-training. This finding indicates that the observed confidence calibration differences between BASE and POST models likely stem from more subtle mechanistic changes that require sophisticated interpretability tools beyond current entropy neuron analysis to fully understand.

We will carefully check the whole paper to ensure each part is clear.

[1] Wes Gurnee, Theo Horsley, Zifan Carl Guo, Tara Rezaei Kheirkhah, Qinyi Sun, Will Hathaway, Neel Nanda, and Dimitris Bertsimas. 2024. Universal neurons in gpt2 language models.

[2] Alessandro Stolfo, Ben Wu, Wes Gurnee, Yonatan Belinkov, Xingyi Song, Mrinmaya Sachan, and Neel Nanda. 2024. Confidence regulation neurons in language models.

Thank you for your detailed and thoughtful feedback and recognition of our holistic evaluation, comprehensive experiments, conclusions, and potential usefulness! We would like to address your concerns and questions respectively as follows.

Reason to reject #1: The experimental setup of the knowledge analyses raises some concerns, and why we did not follow the ROME setting.

We chose this setup from [1] because of the following considerations. First, our setting (e.g., "The city of Toronto is in Canada. This statement is:") can detect knowledge storage in both subject and object. In contrast, the ROME setting [3] provides the subject and lets the model outputs the object, e.g., "The city of Toronto is in". It can only detect knowledge storage in the subject. Second, our setting can test a wider range of factual knowledge. ROME setting evaluates patching's influence to the output logit of the correct objct, so it must have a single correct object, such as the country of a city. We did causal tracing experiments on 7 datasets, but only 2 of them do have a single correct object. They are:

• cities: The statement is "The city of xxx is in xxx", where the object should be the country of the city.
• sp_en_trans: The statement is "The Spanish word 'xxx' means 'xxx'", where the object should be the English translation (although it might have some synonyms, it is almost fixed).

The other 5 datasets do not have a single correct object. They are:

• neg_cities: The statement is "The city of xxx is not in xxx", where the object can be any country except for the correct one.
• larger_than: The statement is "xxx is larger than xxx", where the object can be any number smaller than the subject.
• smaller_than: The statement is "xxx is smaller than xxx", where the object can be any number larger than the subject.
• neg_sp_en-trans: The statement is "The Spanish word 'xxx' does not mean 'xxx'", where the object can be any English word except for the correct translation.
• tulu_extracted: The statements cover a wide range of factual knowledge, such as "The digits of Pi are infinite." and "A car is something you drive, not cook with."

Third, these datasets have “clear scope” and “are simple, uncontroversial, and unambiguous” [1]. So we expect LLMs' output should align well with their knowledge.

Nonetheless, we fully understand your concerns based on [2], which is very reasonable. Following your suggestions, we conduct experiments based on ROME setting [3] on cities and sp_en_trans datasets. In our previous experiments, a pair of statements contain a true statement and a false statement, where they have the same object but different subjects. Now we directly let the model output the object. We denote the model's output object for one statement as $O_1$ and the output for another statement as $O_2$. We use $log \frac{P(O_1)}{P(O_2)}$ to denote whether the patching is successful. We use the same metrics as Table 1 to evaluate the results. The results of Llama-3.1-8B model family are shown below:

The results of Mistral-7B model family are shown below:

The results are even stronger than Table 1. It verifies our conclusion that post-training has little influence on knowledge storage locations. The results on cross-model patching is also simialr to our previous conclusions, though being slightly different. The backward transfer (patching from post-trained model to base model) is slightly available, but not as good as forward transfer (patching from base model to post-trained model). We are not able to upload visualization results in the rebuttal. We will update our paper to show all the new visualization results.

Reason to reject #2: The refusal analyses do not add new insights from what's known in the literature.

We guess you might be referring to paper [6] rather than paper [1]. The key distinction between our paper and [6] lies in our experimental methodology: In Appendix J of paper [6], the authors computed cosine similarities between activations (from base and chat models) and a single refusal direction (extracted from the chat model). Our approach directly computes cosine similarity between the base model's refusal direction and the chat model's refusal direction. Additionally, we conduct transfer intervention experiments to causally validate our findings. In conclusion, while we acknowledge the contribution of paper [6], our own experiment settings are different from it and do provide additional insights into refusal directions in BASE and POST models.

Reason to reject #3: The methods used for analyses are not well-motivated. The paper does not use a consistent set of analytical methods.

We totally agree with you that there are often several different interpretability methods available for analyzing each type of model behavior, and the differences, pros, and cons between them can be subtle. Our rule of thumb is to use the latest and most suitable methods for each of the four perspectives.

We would like to first clarify our analysis techniques. We use causal tracing for analyzing knowledge; probing and intervention (and causal tracing, which is used to select the layer to build the probe) for truthfulness; intervention for refusal; and entropy neurons for confidence. We discuss the subparts one by one in the following.

It is unclear to me why the same probing analyses are not used for refusal

We made such a design because the difference between truthfulness and refusal is subtle. The probing analysis for truthfulness aims to understand if the model can internally distinguish between true and false inputs. However, refusal is a property centered at outputs. The refusal interpretability method we explored in this paper is the best method we know, and it learns the refusal direction from harmful/harmless data. So the learned direction represents an "internal belief of harmfulness", which corresponds to how the model distinguishes a statement being harmful vs harmless, but not necessarily whether the model refuses to answer an input or not. Although they are related, they are not exactly the same. To avoid potential mislead and to directly test refusal behavior, we use the Refusal Score, which is intervention-based and allows us to directly evaluate the refusal results based on outputs. We see this as a more reasonable evaluation design that can be more easily extended to future methods for learning better refusal directions.

In fact, in paper [6]'s Appendix J, their cosine similarity experiment can be considered as a form of direct probing on the input activations (the only difference between cosine similarity and direct probing is that linear probing means a sigmoid function on top of the dot product of the probing direction and input activations). However, due to the gap we discussed above, we believe the evaluation based on Refusal Score is more aligned with what we need.

why causal tracing and probing are not used for analyzing confidence calibration.

Unlike binary knowledge verification tasks, where factual statements can be cleanly categorized as true or false, model confidence exists on a continuous spectrum that cannot be easily categorized into 'confident' and 'not confident' categories. This fundamental difference makes the causal tracing approach unsuitable, as it requires distinct clean and corrupt runs to isolate causal effects. Similarly, differences in mean probing methods depend on having well-defined input pairs representing confident versus non-confident states, which is problematic when confidence is inherently graded rather than categorical. To the best of our knowledge, none of the previous papers have utilized causal tracing or probing to analyze confidence calibration.

We therefore chose to focus on entropy neurons based on established evidence from prior work demonstrating that these neurons function as internal confidence calibrators within the model's architecture. This approach allows us to study confidence mechanisms without artificially imposing binary distinctions on what is naturally a continuous phenomenon.

Prior works have found that there might exist knowledge neurons (KN) [4] and truthful attention head [5] to explain knowledge and truthfulness, respectively.

As we discussed above, our rule of thumb is to use the latest and most suitable methods for each analysis.

Knowledge Neurons (KN), as a great method for knowledge analysis, does not fit our purpose as good as causal tracing. KN was originally proposed as a method on encoder-decoder models, specifically BERT, whereas causal tracing was more recently proposed as a method on decoder-only models explored in this work, e.g., Llama, Mistral, etc.

Moreover, even though KN has been adapted for decoder-only models in subsequent work, empirical evidence from [3] demonstrates that causal tracing outperforms KN during model editing tasks, indicating that causal tracing provides more precise localization of stored knowledge. Given that transfer knowledge editing represents a potential application of our findings, this superior performance makes causal tracing more suitable for our knowledge analysis objectives. Additionally, since our truthfulness analysis follows [1] in employing causal tracing for intervention layer selection, adopting causal tracing for knowledge analysis ensures methodological consistency across our experimental framework.

We chose not to use truthful attention heads as it is also not the most suitable method for our analysis.

First, our experimental design follows [1], which focuses on datasets of "clear, simple, and unambiguous factual statements" to isolate truthfulness mechanisms without confounding factors. This approach differs from [5], which employs datasets such as TruthfulQA that contain intentionally misleading questions designed to test models' susceptibility to common misconceptions—a valuable feature for evaluating robustness but not aligned with our specific experimental objectives.

Second, truthful attention heads have primarily been validated on adversarial or ambiguous datasets, but their effectiveness on our curated setting of straightforward factual statements remains unproven. Since we are adhering to the controlled experimental framework established in [1], we opted for the MM-probe method, which has demonstrated reliable performance in similar settings with clear factual statements.

Reason to reject #4: This paper could be substantially strengthened if it could provide deeper analyses on one of the four phenomena of interest, especially analyses of how different post-training methods have different effects.

We agree with you that doing experiments with different post-training algorithms is very meaningful, while analyzing their effects can be a separate paper for each perspective. For example, paper [7], which we cited in related work, explores how DPO affects the toxicity of the models. Our work aims to provide a systematic analysis of all four perspectives that we believe are important for understanding the post-training effects.

Question 1: Line 152: How do you ensure that "every statement from tulu_extracted also appears in the tulu dataset"? Do you look for the verbatim occurrence of the statement?

There are two Tulu datasets: tulu-3-sft-mixture dataset, which was used to finetune Llama-3.1-8B-SFT, and tulu-v2-sft-mixture, which was used to finetune the Mistral-SFT model. flan_v2_converted is a common subset of tulu-3 and tulu-v2. We use GPT-4o to extract many factual knowledge statements from flan_v2_converted. We manually picked 100 factual knowledge statements from the generated statements that are high-quality, valid factual knowledge, and indeed appear in the flan_v2_converted dataset. Therefore, these factual knowledge statements must exist in the fine-tuning dataset of Llama-3.1-8B-SFT and Mistral-SFT models. While we reformatted the syntax to maintain consistency with other datasets, the underlying content and knowledge remain intact. We have recorded the indices mapping each tulu_extracted datapoint to its location in the tulu-3-sft-mixture dataset. To ensure full transparency and reproducibility, we will include these mapping indices in our upcoming code release.

Question 2: Is there a reason for choosing this pair of scalars? Would the conclusions be different if the scalars are tuned on a validation set of better intervention effects?

We are following the original scalar setup in [1], where the scaling factors are constrained to +1 or -1. However, we acknowledge that prior works on model steering for truthfulness [5] have demonstrated that scalar choice can impact intervention effectiveness. To directly address this concern and ensure our conclusions are robust across different scaling configurations, we conducted additional experiments varying the scaling factor from 1 to 10 with a step size of 1 on the llama-3.1-8b and llama-3.1-8b-instruct model pair.

For each dataset and model combination, we identified the optimal scaling factor that maximized the performance. We report the optimal scaling factors in the table below. INS→INS means using the truthfulness direction of the instruction-tuned model to intervene in the instruction-tuned model. BASE->INS means using the truthfulness direction of the base model to intervene in the instruction-tuned model. Coef records the optimal scaling factor, and IE (intervention effect) is our metric evaluating the intervention as in Table 3. Delta is the difference between BASE→INS IE and INS→INS IE.

The results show that base direction and instruct direction have similar intervention performance when both use their own best-performing scalar choice. Although the intervention results are a little sensitive to the scaling factors, our main conclusions remain valid even when scaling factors are tuned: both the base direction and instruct direction can be used to intervene on the instruct model with equal effectiveness.

Reference:

[1] Samuel Marks and Max Tegmark. 2024. The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets.

[2] Xiang Lisa Li, Vaishnavi Shrivastava, Siyan Li, Tatsunori Hashimoto and Percy Liang. 2024. “Benchmarking and Improving Generator-Validator Consistency of Language Models.

[3] Kevin Meng, David Bau, Alex J Andonian, Yonatan Belinkov. 2022. Locating and Editing Factual Associations in GPT.

[4] Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, Furu Wei. 2022. Knowledge Neurons in Pretrained Transformers.

[5] Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, Martin Wattenberg. 2023. Inference-Time Intervention: Eliciting Truthful Answers from a Language Model

[6] Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. 2024. Refusal in language models is mediated by a single direction.

[7] Andrew Lee, Xiaoyan Bai, Itamar Pres, Martin Wattenberg, Jonathan K. Kummerfeld, and Rada Mihalcea. 2024. A mechanistic understanding of alignment algorithms: A case study on dpo and toxicity.