Truth-value judgment in language models: belief directions are context sensitive

Thank you for your feedback.

We limit our investigation to these methods, because we believe them to be representative of methods used in the existing literature. It may be right that this is only a small portion of the space of all possible models for this problem. However, we consider developing entirely new methods to be out of scope for this work.

In response to your questions:

1. We tried to be careful in our language use. We specifically say ‘predictive of the truth’ and ‘represents as true’, rather than just truth or true. We chose ‘truth-value judgment’, where ‘judgment’ implies judging (subjective), and ‘truth-values’ emphasizes that we mean the truth of sentences, rather than some grander Truth. So we definitely did not mean to imply that LLMs have access to anything like objective or actual truths. If there are any specific places in the paper where we could do better on this; or specific phrasings you think we should avoid, we would greatly appreciate your suggestions.
2. Marks & Tegmark visualize the result of a causal experiment over the layers. We believe there is more structure there because those experiments are based on interventions. Interventions might show more structure, because an intervention will probably have some effect. And, if the effect is not the intended effect, it will probably be some random effect, and those are likely easy to tell apart. Probing experiments are more diffuse, because even if information stops having an effect on the output by layer 15 (for example), it can still ‘dormantly’ remain present up to layer 35, and thus still be picked up by a probe.
3. Yes, this is a good point. Following Marks & Tegmark we will also report the ‘normalized indirect effect’ (NIE). This quantifies the effect of the intervention proportional to the difference between the observational probabilities: $\frac{p(\mathbf{h}; do(\mathbf{q}^+{\mathrel{-}=}\mathbf{\theta})) - p(\mathbf{h}; q^+)}{p(\mathbf{h}; q^-) - p(\mathbf{h}; q^+)}$. An NIE of 1 indicates the intervention can account for the full difference between true and false. This will allow for a fairer direct comparison.

Thank you for your feedback.

You rightly point out that it is not a given that there exists a probability distribution $P_\lambda(X)$, or that we can learn it with linear probes. We do make such an assumption, but we believe this assumption is reasonable. First, previous work suggests such linear probes (c.q. directions) exist, we will clarify this (near line 142) by adding references to papers that find such directions. Second, we do report the probe accuracies in Table 2. Accuracies are generally high, suggesting that we are finding directions at least correlated with the truth. Finally, our method does not merely assume that the probes actually identify beliefs, rather, our methodology also provides a way to test for another property we can reasonably expect beliefs to have, which is that they should be updated in light of new information. Of course, a failure to observe this does not rule out the presence of beliefs (as they could be prior beliefs), but observing that the probabilities depend on in-context information does further support calling such probabilities beliefs.

Given the high accuracies, although it is still possible that the probes are (partially) identifying other features (such as sentiment), it would require those features to be highly correlated with the truth of the sentences. However, we specifically set up our experiment such that each sentence in the original dataset occurs both affirmed and negated, meaning for each pair, almost all features except sentence-truth are shared between them.

We agree that the performance of the pos-prem probes is likely superior because of the evaluation data being in-distribution. However, we do not believe that detracts from our results. The fact that different training distributions produce different probes, that are often still close in accuracy but not in sensitivity, suggests that there might be multiple directions in latent space that could be belief directions. Our error scores provide a way to characterize the differences between probes that show similar accuracies.

In response to your questions:

1. • $\lambda$ is our way of marking this probability as different from the model’s probability over tokens (we use $\lambda$ as short for latent). We will clarify this in the paper.
   • X is defined on line 141
   • $\sigma$ is the sigmoid function, we will clarify this.
   • Yes, $\mathbf{\theta}$ on line 162 should be a unit vector, we will clarify this.
   • $Q^+$ and $Q^-$ should be defined similarly to $X^+$ and $X^-$ (line 137), we will clarify this.
2. There is no bias term, because the hidden states have their mean subtracted from them, and we assume (as previous work did) that this makes a bias effectively redundant, we will explicitly state this.
3. As we mention in lines 168-172, we found CCS to be relatively inconsistent in its performance. Other methods identify slightly different directions from one layer to the next, but CCS does not, it can vary considerably. This makes CCS harder to analyze. And, because we still wanted to have unsupervised methods among the results, we introduce CCR in this paper. We will state the latter explicitly in the paper as well.
4. While one might indeed want to specifically find a conditional belief direction, the goal of the present work was to investigate the way existing methods interact with the context. We develop ways to measure different types of context-dependence, but leave the training of probes that specifically exhibit (various kinds of) context sensitivity for future work.
5. By mean-normalized we express that the vectors have their mean subtracted from them: $\mathbf{x}^+ = \mathbf{x}^+ -\mathbf{\mu} \text{ and } \mathbf{x}^- = \mathbf{x}^- -\mathbf{\mu}, \text{ where } \mathbf{\mu} = \frac{1}{2|\mathcal{D}|}\sum_{(\mathbf{x}^+, \mathbf{x}^-) \in \mathcal{D}} (\mathbf{x}^+ + \mathbf{x}^-)$, we will specify this in the paper.
6. The average probability assigned to the hypotheses without any premise is indeed around 0.5. This is because of the mean normalization, and because we have equally many true and false statements. The mean normalization ensures that the average projection onto the belief direction is around 0, which causes an average output of 0.5 after the sigmoid.
7. We will add a table of variances to the appendix.
8. The premise sensitivity is defined as the mean absolute premise effect (line 242), where the premise effect is $p(\mathbf{h}; q+) − p(\mathbf{h})$ (line 241).

Thank you for the reply.

We are sorry to hear aspects of the paper feel underdeveloped. In your review you praised the importance of our main question: 'whether belief directions depend on context', is there a particular angle you think could strengthen our motivation, or another aspect of the motivation we could improve (in addition to what we clarified in response to the other reviews)?

Regarding the specific concern of sentiment. Looking at the data samples we provide in the appendix, we can see that the datasets we use contain (near-)exclusively sentences with no or neutral sentiment. So it really seems an unimportant feature in our data and there is no indication that it might be correlated with truth.

We do acknowledge the broader concern of features potentially being correlated with truth, which our probes might learn instead of a genuine truth-feature. As mentioned previously we believe we have mitigated it by the way we construct pairs of negated and affirmed sentences. Furthermore, this issue is not unique to our setting, this risk exists in all of the experiments of the works we cite. In our work, we set out to train belief probes in a way similar to previous work, and then evaluate it in novel ways. Furthermore, there are probably many thousands of features that a probe might exploit to improve its accuracy. If we do not see an explicit reason to suspect any particular one of them, then how do we decide which to test for?

In response to your questions:

• On line 141, we define bold $\mathbf{x}$, which is the vector extracted from the LLM's latent space. On line 145, we indicate that we use a unbolded italic lowercase $x$ as a shorthand for the variable $X$ being equal to 1. $X$ denotes the truth-value of a sentence, which could be 1 (true), or 0 (false).
• Here we follow previous work (Burns et al. 2023; Marks & Tegmark 2024), which favored centering the data over figuring out how to learn/find an appropriate bias term.

Thank you for your feedback.

To improve the discussion of the E3/E4 scores, we will add an example. We will alter section 3.3 to include the following content:

Take for example Q=“December is during the winter for New York” and H=”In New York, days are shortest in December”. 
If the model assumes the context is true when determining the probability for H, then:
  (1) having the context say ‘Q is incorrect’ should decrease the probability of H, or 
  (2) having the context say ‘Q is correct’ should increase the probability of H.
This expectation is captured by E3.

If however the model only pays attention to its own evaluation of the truth of Q, then having ‘Q is incorrect’ or ‘Q is correct’ should not influence the probability of H at all. In that case, the presence of Q merely acts as a reminder that the truth or falsity of Q is something the model should consider when evaluating H. This expectation is captured by E4.

In response to your questions:

1. By inconsistency hallucination we mean inconsistency between an LLM’s generated output and: (1) the instructions, (2) additional information that is provided in context, or (3) logical inconsistency with another part of the generated output.
2. Yes, we will add more comparisons between the two in the appendix, to support this claim further.
3. We use the affirmative premises, because those are the variants for which we have the clearest expectation of what should happen. Affirming the premise preserves the original meaning relation (entailment or contradiction) to the hypothesis, which we know. In contrast, negating the premise can change the meaning relation to the opposite or to neutral. In the latter case we do not expect the probability for the hypothesis to change w.r.t. the no-premise setting.
4. Yes, this is a possible benefit of using unsupervised methods. We think this is another good reason to care about the in-context behavior of belief probes. We might identify directions that, despite being less accurate, still do well at truth-value judgment, meaning the probabilities update appropriately in response to additional information. We would have good reason to still think of those as beliefs, despite them disagreeing with what we consider true.

We would like to ask if clarifying what we mean by inconsistency hallucination (and adjusting the paper accordingly) is sufficient to clear up the motivation. Or do you think the motivation be revised more extensively anyway?