Personas as a way to Model Truthfulness in Language Models

Thank you for your time and the constructive feedback!

I’m confused by the use of “agent”, “persona”, and “truthfulness” in the paper.

Please, see “terminology” in our general response, we have clarified the choice of the terms in the paper. We wanted to make the distinction between an agent (as defined in Andreas 2022) and a persona. We’ll make sure to highlight and more clearly state the definitions in our revised version of the paper.

Probe through the answer generation process. It will tell us that the persona the model puts on is also formed during the decoding process stochastically.

Thanks for the suggestion, this is definitely an interesting experiment worth adding to the final version! We ran the experiment and found that the F1 score indeed increases to 70% when the entire answer is sampled. We have updated the PDF with the new figure in Appendix B (figure 9).

The fine-tuning experiments aren’t informative/surprising. We have known this from inference-time intervention (Li et al. 2023).

Firstly, we note two important distinctions from the experiments in Li et al. ---

1. We test the model’s OOD performance by holding out certain categories of questions during finetuning. A generalization to unseen categories is stronger evidence supporting our hypothesis.

2. Just evaluating on truthfulQA leaves up on the possibility of model picking on artifacts on truthfulQA (e.g. the dataset has a lot of false presupposition and `no’ is often the correct answer). We showed that the finetuning also generalizes to a separately created dataset from BigBench. We also want to emphasize that our goal with the finetuning experiment was to demonstrate how it could be explained by our hypothesis (hence the held-out experiments) rather than demonstrating a new method to improve truthfulness.

Most importantly, I’m afraid this submission lacks novelty. Specifically, given Andreas 2022 proposing a wider hypothesis, Burns et al. 2022 show truthfulness could be probed from internal activations, Li et al. 2023 show there is a simple trick to boost the truthfulness of LLM.

Thanks for bringing this up! We meant this work to be a bridge between Andreas and {Li et al., Burns et al.} where we show how the findings from the first can help explain the observations in the latter. We wanted to note a few points regarding novelty:

1. Prior work demonstrates that models have learnt some concept of truthfulness but none of these works gives even potential mechanisms of models learning truthfulness. We present the “persona” hypothesis, and show evidence for how LLMs can model truthfulness via modeling personas.
2. The fact that LLMs learn truthfulness is surprising and begs the question of where the capability comes from. It’s not consistent with the classic view of LMs (which many people in the community may hold), which suggests that the model should generate true/false answers according to their frequency in the data but not necessarily encode truthfulness and falsehood (which has no explicit label in the data). It is also surprising from the supervised learning perspective that models generalize truthfulness from one topic to another without knowledge transfer. This work fills the gap by providing an explanation.
3. Even though the connection might seem obvious to some people, we still think there is value in more concretely demonstrating it. Particularly, our synthetic experiments allow us to create a counterfactual scenario (what if there was no truthful persona in the data) where we demonstrate that we do not see the probing and finetuning observations as on real LLMs — a stronger evidence for the persona hypothesis.

Could you provide other metrics, like accuracy, which is more widely adopted in probing literature?

The training data is balanced 50/50 but the test set is imbalanced — it has a lot more examples where the model is untruthful than it is truthful (this is just generally true for LLaMa which only gets about 35-40% truthful on truthfulQA). This is why we reported the F1 score weighted by class size. The random guessing baseline is not 50% F1 score but < 50% F1 (~46% F1 score) in this case. Based on your suggestion, we also compute accuracy which we find to be very similar to F1 because of the balanced training data. We included this figure in Appendix B (figure 8).

Thank you for your time and the constructive feedback! We address some of the concerns here and some general comments in the ‘general response’:

Reliance on appendices for crucial information

For the LLM experiments (section 2), we give all the essential information such as experimental setup and evaluation criteria in the main paper — we only included the exact prompts and the hyperparameter details in the appendix. For the synthetic experiments, we did not include the exact definition of the operators in the main paper (Appendix D.1 and D.2) since the important part we wanted to convey here was the behavior of the agents, e.g. when they are truthful. Let us know if something among these was crucial and you find it helpful to include in the main paper instead of appendix! We are happy to address it.

Reliance on appendices may create doubts about the robustness of the synthetic data experiments.

As mentioned before, we only included the exact definition of the operators in the appendix, we believe our main findings do not depend on these. Additionally, for all experiments here (sec 3.1 and sec 3.2) we show the variance of results across different semantics of the operators — this further reduces the reliance on what the exact definition of each operator is. Please, let us know which details could be included in the main text to improve the trustworthiness of the results.

Insufficient comparison with other methods and a lack of an ablation study.

Firstly, we want to clarify that we did not propose any new method but rather proposed a new hypothesis to explain how LLMs could model truthfulness. All our experiments are to show evidence for this hypothesis. We also did ablations for probing to understand how performance varies across layers and across different prompt tokens (Appendix B), and the synthetic setup (setting with no truthful persona in Figure 4 and Figure 5). If there is some other ablation you would like to see, we are happy to include it!

The paper is hypothesis-driven and lacks comparative experiments with existing methods.

Indeed the paper is hypothesis driven! Our goal here is to better understand how LLMs can learn abstract concepts like truthfulness rather than to propose new methods. As such, we are not sure which relevant existing methods it would be useful to compare to, since we do not propose a new method but rather a hypothesis of how LLMs can model truthfulness. But again we are happy to include if you had something specific in mind!

Lack of generalization experiments

We do show for some experiments that our findings generalize to an unseen dataset (BigBench misconceptions). We also conducted experiments to make sure that the model is not relying on specific TruthfulQA artifacts (e.g. by having held-out categories). We use all data sources available, to the best of our knowledge, and ablate them for generalization experiments.

Better explain the the intersection of TruthfulQA and TriviaQA, or TruthfulQA and MS MARCO.

Thanks for raising this point. We already include these comparisons in the current paper, but we will clarify in our revised version. We use TriviaQA and MS Marco to make sure that the results we obtain by finetuning on truthfulQA are not due to the model simply getting better at question-answering. We chose these datasets since the model can still learn general question-answering abilities from them, but these questions do not have common untruthful answers in the (pretraining) data (i.e. they will not provide training signal for truthfulness). Thus, the model could improve its QA abilities but not update its representations of personas. Ablating this shows that finetuning on truthfulQA is successful by eliciting the truthful persona rather than learning general QA abilities.

Thank you for your time to review our paper and to give constructive feedback. We also suggest you go through our general response first since it clarifies our terminology and addresses general issues different reviewers mention.

The truth and the falsehood in training data of LLMs should be given formal and clear definitions.

Thanks for bringing this up. We have clarified in the general response what we mean by truthful and untruthful text. Specifically, truthful text contains factually correct information —- our focus is on factual questions where there is a widely agreed upon answer (e.g. experts would agree). Untruthful text contains incorrect (i.e. not grounded in evidence) but plausible answers — these are answers which are supported by misinformation in the pretraining data. As an example (from the updated Figure 1), "covid vaccines are ... extremely safe" is truthful text, whereas the text “...millions die from covid injections” is untruthful since it is incorrect but is probably seen in the pretraining data since it is a misconception generated by a source ("Before It's News" in this case).

The authors only leveraged additional model to classify the results, more evaluation strategies should be considered.

We interpreted this comment in two ways:

(a) ‘additional model’ refers to the probe used to classify true vs false statements; We use probing to predict if the generation will be truthful only from the representation of the question. This means that we can linearly separate truthful and untruthful text generated from different agents — evidence which is consistent with our hypothesis. Probing is an established empirical method to evaluate the presence of representations in the latent space of a model.

(b) ‘additional model’ refers to using GPT-judge as the evaluator of truthfulness. We want to note here that in addition to using GPT-judge (model to evaluate truthfulness), we also do human evaluation for both the datasets (truthfulQA and BigBench-misconceptions) in Sec 2.2. Additionally, we note that there is a strong correlation between both forms of evaluation (model based and human) — both as observed in our work and as reported in previous work (e.g. Lin et al 2021).

Thank you for your time to review our paper and constructive feedback! We suggest you go through our general response first since it clarifies our terminology and addresses general issues different reviewers mentioned.

First of all, the terminology "persona" -- my thinking around persona has been mostly influenced by Social Simulacra [1], using constructed persona profiles as prompts for LLMs and controlling what the LLMs generate.

In our general response, we have motivated the choice of the terms in the paper. We chose the terms ‘agent’ to be consistent with the work we build on (Andreas et al), and ‘personas’ as a natural extension of the term.

from Figure 1, it is hard to tell what is the "truthful" or "untruthful" persona the paper is referring to.

We have updated Figure 1 to reflect what is a "truthful" or "untruthful" persona --- it refers to a latent variable corresponding to a cluster of agents (left part of figure), which the model can use at inference time to generalize truthfully or untruthfully (right side of the figure).

Another way to understand “persona” is that truthful text shares some common features — we didn’t want to call it `features’ because it can imply that these common properties are local (e.g. using specific words), whereas they could be more abstract properties. More generally, our hypothesis is that agents which are truthful (e.g. Wikipedia, Science journal etc.) tend to form a cluster due to common properties like consistency, providing evidence etc. — we refer to this cluster of truthful agents as the truthful persona.

The paper would be hugely improved if authors could clearly define what they mean by agent and by persona.

We have addressed this in our general response and will clarify in a revised version of the paper.

LLMs model persona and each persona is an agent?

We elaborate on this in our general response. The short answer is that each persona is not an agent; a persona is a group of agents that share common properties like truthfulness. Both agent and persona are used to explain the data generating process.

Where are the personas defined in the models?

It is important to highlight that agents and personas are constructs to model the data generating process (i.e. latent variables), and they do not necessarily correspond to physical entities. However, we find evidence that these constructs are useful to explain model behavior. We use our probing results to show that we can decode a “truthful persona” from the latent activations of the model, which suggests the model encodes this information in its hidden states.

The synthetic setup is not close to reality

We want to clarify a potential misunderstanding in the Sec 3.1 setting. In the first setup, when the two agents (C and D) are fully dishonest, we mean that the probability that the agents are truthful on each operator is small (< 0.2 — sec 3 dataset generation & appendix D.1). Therefore the agents are not always untruthful but they do have a small probability of being truthful. Similarly agents A and B are not fully truthful, but have a small probability of generating an untruthful answer. We think this is a realistic setup, where there is no one source which is 100% truthful or 100% untruthful —- but some sources are much more likely to be truthful and trustworthy (e.g. Wikipedia would have a high probability of being truthful ; Similarly an author who believes in a conspiracy would mostly write untruthful statements to support the conspiracy but that will be mixed in with some amount of truthful information).

Would it be more interesting if the authors could extract these personas or show how fine-tuning or ICL can actually force LLMs to always adopt a truthful persona

In our experiments (sec 2.1) we extract these personas by training linear probes to distinguish between true and false statements based on hidden states from the model. This is evidence consistent with our hypothesis that LLMs infer persona from context and represent it in the hidden states, which the probe can rely on, i.e. the probe is extracting the persona. However, we do not have interpretable features that represent the persona. Also, we show that both finetuning and ICL steer the model to adopt the truthful persona (sec 2.2).

References

[1] Andreas, J. (2022). Language Models as Agent Models. ArXiv, abs/2212.01681.

[2] Park, Joon Sung, et al. "Social simulacra: Creating populated prototypes for social computing systems." Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology. 2022.

[3] Burns, Collin, et al. "Discovering latent knowledge in language models without supervision." arXiv preprint arXiv:2212.03827(2022).

[4] Li, Kenneth, et al. "Inference-Time Intervention: Eliciting Truthful Answers from a Language Model." arXiv preprint arXiv:2306.03341 (2023).