Inspection and Control of Self-Generated-Text Recognition Ability in Llama3-8b-Instruct

We thank the reviewer for their thoughtful comments. We agree that the question of how the styles of human and LLM languages differ is a fascinating and important one. It’s something we’d like to explore further in future work; in this work, we’ve focused on explaining the differences that this model and this vector are picking up on. In Part 1, testing model self-recognition ability, we find that the model does use length in the paired paradigm (unless we normalize it). We did do token-level analysis of vector activations (not included in the paper), but didn’t find anything that stood out, likely because the tasks in our paradigms forced more similarity in word choice than would occur in more naturalistic settings (e.g., summarizing an article about X will require X-related words no matter who is writing it). However, as can be seen in appendix A.4, the vector does show higher-level preferences for a more positive tone and a less technical writing style. Qualitatively, when steering with the vector, the model’s responses become more cheery and optimistic, even inappropriately so when steering with a high enough multiplier.

We have added details to the figure captions to make them clearer.

Regarding the question about the U-shaped effect of steering: this is not due to a reversal of judgment but rather to higher multipliers leading to degenerate (disfluent or random) outputs, and this effect is stronger at earlier layers (which have lower norms). (The finding of a U-shaped effect is not unique to our work see, e.g., this work from Anthropic: https://www.anthropic.com/research/evaluating-feature-steering). Figure 4 shows negative effectiveness because effectiveness is expressed relative to the unsteered model. We have added clarifying text.

We thank the reviewer for their comments. Regarding focusing on one model family, we acknowledge this limitation and plan to explore others in future work; however we believe we have established an important existence proof. We expect that the findings regarding self-recognition ability will apply to larger models and ones outside the Llama family. Prior work (Panickssery et al 2024, Laine et al 2024) using similar paradigms suggests that RLHF’d Claude and GPT models show comparable self-text recognition abilities to Llama ones, and that abilities increase with scale, and our own experiments with Sonnet 3.5 using our current paradigm (now shown in Figure 14, which we’ve added to the appendix) indicate that is has superior self-text recognition abilities to Llama3-8b. We now address this point in the Discussion section.

While the specific vector we utilize here will not extend to models with different dimensionality and layer count, we expect that the process to identify and use the vector will be transferable, since prior work has successfully used the contrastive prompts paradigm and residual stream steering in a variety of model sizes. We look forward to applying this process and exploring the basis of self-text recognition in larger models in future work.

We acknowledge the point about cross-referencing figures in the appendix, however the paper page limit constrains our flexibility in this regard.

Regarding Question 2: we have added details on vector creation to the Methods section.

Regarding Question 3: we do not employ fine tuning in this work. If the question is about how robust the vector is to different datasets, we have conducted additional experiments forming the vector separately from contrastive pairs for the CNN, XSUM, DOLLY, and SAD datasets. We find that the vector is quite similar across the first three (cosine similarity in the .85-.90 range). The similarity is lower between those and the SAD dataset (~.5); however, we believe that the similarity is driven by the higher-level, semantic properties of the vector, while the differences are driven by idiosyncrasies of the text in the different datasets (the first three all being stylized short summaries, while the SAD dataset is longer and more diverse). When we take the average of the SAD vector and a vector from any of the summary datasets and inspect its token mappings with Tuned Lens, we see self/other concept tokens (“I”, “my”, “someone”, “other”), just as we do with the originally identified vector. We have added section A.12 to the appendix to reflect this and called it out in the Methods section.

We have added text on implications for model safety to the Discussion section.

We thank the reviewer for their thoughtful comments. We agree with the reviewer that the vector is carrying information about what text generated by an RLHF’d model looks like, rather than information about how its output differs from that of different RLHF’d models (we call this out and discuss it in lines 181-187). Indeed, it’s not clear how the model could have learned to make the latter distinction, as the Llama3 model card[1] does not describe any process by which the model would have been exposed to labeled output from other models during pre- or post-training. We believe that the model’s ability to distinguish RLHF’d model output from that of humans is noteworthy, and is relevant to the safety risks we mention: it could help the model collude with other LLMs (either copies of itself or different models) and detect when it is interacting with humans. We also note that the vector is not solely carrying information about characteristics of RLHF’d output; as shown in our open-ended generation experiments (appendices A.3 and A.9), steering with the vector on prompts unrelated to text recognition induces or inhibits output related to the concept of “self”.

Regarding including the source text and instructions in the self-recognition prompt: this was motivated by 1) consistency with prior work, and 2) desire to allow the model to base its judgments on factors besides just textual style (e.g., which information it would choose to extract from a text in order to create a summary of it). We also expected that, given the short length (1-4 brief sentences, <100 tokens) and stylized nature of the summaries, it would be nearly impossible for any model (or human) to tell whether it wrote them without some knowledge of the context. Inspired by the your comment, we went back and tested this on the individual recognition paradigm. Without at least some sort of contextual information (the article or the instruction of how to formulate a response), the model cannot distinguish self- from human-written text in the summarization datasets. But on the longer (approx 400 tokens), more naturalistic continuation and QA datasets, the model performs well above chance(62-65%) with no additional context (no text starts in the continuation dataset, no questions in the QA dataset, and no explanation about where the text to be judged came from in either). We have added section A.11 to the appendix to show this and call it out in the Methods section.

We have added a summary to Section 3.2

Regarding line 151 and Figure 1: Figure 1a is the "unnormalized" text, and SAD is at bottom right. We have made the font larger and called this out in text.

Regarding Table 1: it only shows the comparison to human text due to the findings in the Paired paradigm, as mentioned in lines 186-187.

Regarding Figure 4 and Table 3: we have edited the legends for clarity.

Regarding Question 2, we trimmed to equal character length; we have clarified in the text.

[1] Dubey, et al. "The Llama 3 Herd of Models." arXiv preprint arXiv:2407.21783 (2024)

I appreciate the rebuttal for engaging directly with my earlier feedback and for making various improvements to the paper.

On the first paragraph of your rebuttal: You call this out in lines 181-187, but elsewhere you describe this as a "self-recognition vector". You sometimes use scare quotes but that doesn't matter from my perspective. Regarding the open-ended generation experiments (as well as appendix A4), I do not find this to be compelling evidence. It does not seem like a systematic quantitative experiment with some kind of "blinding" from your hypotheses (as in a double-blind experiment). I believe that the vector you have found is interesting but you have not characterized how it functions with systematic evidence.

Indeed, it’s not clear how the model could have learned to make the latter distinction, as the Llama3 model card[1] does not describe any process by which the model would have been exposed to labeled output from other models during pre- or post-training.

I do not understand the point here. Presumably, there are generations from many LLMs in the pretraining data and these would often include the name of the LLM. Moreover, even if the name of the LLM is not always present, the model could still make use of this "unlabeled data" (as in semi-supervised learning). Note that you could also investigate fully open-source models such as Olmo/Tulu 3, in which pretraining and full post-training datasets are made available.

This was my most important weakness and I feel it has not been adequately addressed. I think the paper is promising but needs more work (and more careful presentation) before publication. Hence I keep my original score.

We thank the reviewer for their perceptive comments. We have attempted to clarify the vector descriptions in the text. We expect that the findings regarding self-recognition ability would apply to larger models and ones outside the Llama family. Prior work (Panickssery et al 2024, Laine et al 2024) using similar paradigms suggests that RLHF’d Claude and GPT models show comparable self-text recognition abilities to Llama ones, and that abilities increase with scale, and our own experiments with Sonnet 3.5 using our current paradigm (see new Figure 14 in Appendix A.10) indicate that it has superior self-text recognition abilities to Llama3-8b. We have added a note on this in the Discussion section. We also find that the vector identified in Llama3-8b-Instruct, when decoded using the Base model’s embedding layer with Logit Lens, shows a good deal of semantic overlap with the Instruct model’s decoding, and a vector identified in the Base model using the same process shows high cosine similarity with the one identified in the Instruct model. We have not directly tested the vector identification and use process with other models, but based on the success with steering vectors reported in the literature with a variety of models, we suspect it would work on those as well. However, this remains an important area of future research.

We acknowledge the point about important information being in the appendix, but the 10-page length limit constrains our flexibility in this regard.

Regarding the question about whether the vector can influence responses to completely out-of-domain prompts, we feel confident in answering in the affirmative. Appendix A.3 (briefly excerpted in section 3.2.3 of the main text) shows open-ended generation responses under steering to questions about authorship of arbitrary quotes, where positive steering leads the model to implausibly claim authorship and negative steering or vector zeroing leads it to implausibly deny authorship. Further, even when the question is not about authorship per se, such vector manipulations can be seen to make the model more or less likely to insert a self reference into a response. Appendix A.9 contains further examples of responses under steering to a variety of prompts. It is also apparent in our subsequent work with the vector on a very different paradigm (in preparation) that steering with the vector causes a particular “style” of output.

Regarding the question of how the self-recognition vectors can be used to mitigate risks, we hypothesize that the vector can be used to to prevent users from injecting realistic fake previous responses, as in the case of many-shot jailbreaks[1], by adding it to the model’s output tokens and subtracting it from input tokens. We also expect that it can be used as a sort of warning system: observing endogenous vector activation to arbitrary text can be an alert that the model knows (or does not know) that it or a human is speaking. We have added these points to the Discussion section.

[1] Anil, Cem, et al. "Many-shot Jailbreaking." The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.