Out-Of-Context and Out-Of-Scope: Subliminal Priming for Large Language Models

We thank the reviewer for the valuable criticism, comments, and questions, which helped us improve several points in the revised version of our paper. Below, we address all points of critique separately and provide answers to the reviewer's questions.

Although this paper is well-executed and sound, the contribution is a bit weak. Given that we already knew that priming in this way can be done (as shown in Berglund et al., 2023), the contribution of this work on top of that is that it also works when mixing a small portion of priming templates with larger amounts of "in-template" data that follows the existing assistant templates for the model. Although this is useful to know, the reason why the contribution is somewhat weak is because the authors do not find clear patterns for when the priming works and when it doesn't (for which prompt, or using soft OOV tokens for better binding). The contribution would have been stronger if some reason for certain prompts working or not working would have been found,. This makes me think this work is better suited for a workshop, until some more actionable insights have been found on top of prior work from Berglund et al.

Let us clarify: our core contributions over existing work, like the article by Berglund et al. (2023), are:

(i) demonstrating that comparatively few descriptions are sufficient to embed OOCR triggers into models (which we motivate theoretically in Section 3), even when providing these outside the model-dependent chat templates, using only a single pass over the data and NOT using auxiliary demonstrations as in Berglund et al. (2023), which we describe as "organic" OOCR (since the model was never shown OOCR examples that it could just learn to mimic),

(ii) showing that OOCR can, in several cases, be triggered when using what we refer to as projective and associative prompts, while prompts like questions fail, even when augmented by a Chain-of-Thought initiator,

(iii) highlighting that soft OOV tokens can improve the embedding/triggering of OOCR similar to conditioning learned representations on special stimuli that are tokenized consistently (hence, acting as anchors).

Consequently, our work allows the following two actionable insights:

(i) OOCR, which can be considered as a primer for situational awareness, can be embedded into models during instruction tuning with only a few manipulated training examples, even for complex behaviours like responding in a different language or with a physics formula/cake recipe.

(ii) Proving the existence of a specific OOCR behaviour can heavily depend on the specific prompting strategy, showing that OOCR may already be much more present in current LLMs than expected.

These two combined show that OOCR may be much more present in any model but not become apparent when using the wrong prompting strategy as a trigger. In other words, we show that the absence of proof is not at all the proof of absence when it comes to OOCR, which can entail severe security hazards (in the form of embedded behaviour outside the spectrum of behaviours models are currently prompted for). This could not be deduced from the results in Berglund et al. (2023), mainly because of their use of auxiliary demonstrations.

The revised version of the paper clarifies these points in the introduction and the conclusion.

When using LLMs as evaluators (and heuristic based overlap evaluators), it's important to verify at least a few outputs manually. Can be a handful randomly selected ones.

We manually verified outputs in all cases and added several illustrative examples in Appendix B.2 for all tested OOCR behaviours. Were the given examples insufficient and should we add more? The revised version of the paper also lists three new OOCR examples in the main body (see Fig. 5).

I can follow the paper with some effort and referring to the appendix, but it can really use some work on clarity. For example, I only understood after reading the prompts in A.6.1. how the two-hop reasoning works. Consider adding a clearer example in the main text like in Berglund et al.

The revised version of the paper lists several examples of descriptions (see “Assistant Data and Test Cases:” in Section 4) and prompting strategies (see Fig. 3). Furthermore, we compressed our experimental analysis to focus on the key points (see Section 5) and moved the details about the remaining experiments to Appendix B.1.

We thank the reviewer for the valuable criticism and comments, which helped us to improve several points in the revised version of our paper. Below, we address all points of critique separately.

The reported work uses existing LLMs and finetuning scripts; the technical innovation is limited to some variants of a previously published prompting strategy, and simple interventions in the finetuning data (composing sentences, replacing some characters in the spelling of names).

We only use minor modifications of fine-tuning cycles, training data and test prompts to show that such small changes are already sufficient to embed and elicit Out-Of-Context-Reasoning (OOCR), which is the primary goal of our work. We do, however, also provide a mathematical explanation for why embedding OOCR with very few primes (descriptions of a response behaviour) is possible, and we show that OOCR may be more present than expected in concurrent iterations of LLMs (because triggering it can heavily depend on the prompting technique). In contrast to the reviewer’s critique, we see this technical simplicity not as a weakness but as a strength of our work, showing that OOCR (which can lead to safety hazards) can be achieved with so few technical modifications as in our experiments.

The value of the work should thus come entirely from revealing novel behaviors in the studied LLMs.

We argue that we achieved this by demonstrating that OOCR can be embedded with few stimuli and triggered by projective and associative prompts, where “normal” prompts (such as unrelated questions) fail, even when augmented with Chain-Of-Thought initiators.

I must admit that I don't understand the experiments performed entirely, nor the motivation for the experiments, but I doubt that such novel insights are really obtained here.

We would be grateful if the reviewer could specify which aspects of the experiments were unclear, so that we can improve this in the paper. The motivation for our experiments was to mimic human subliminal priming studies in the context of OOCR, where the response behaviour of large language models (LLMs) can be manipulated indirectly (line 40 of the previous version). This “indirect manipulation of behaviour” is the bridge between OOCR and (human) subliminal priming, and we investigated whether OOCR can be embedded and triggered with a few priming stimuli. Our experiments show that this is indeed possible in several cases, although triggering OOCR can depend on the prompting strategy. Both of these are novel insights into the emerging field of OOCR. Additionally, we show that soft OOV tokens can act as context anchors (similar to conditioning a response behaviour on specific tokens) and provide a theoretical explanation for why a few primes can influence the learned representations of LLMs (which we further corroborate with the empirical results displayed in Figs. 4 and 5 of the previous version). The revised version of the paper clarifies in the introduction and the conclusion this novelty of our results and their relevance . To make everything more intuitive, we also reduced the technical details in several places (moved to the Appendix) and instead went for an example-heavier approach in Sections 4 and 5.

The paper is written in a confusing way, that mixes motivation and description of the finetuning and prompts (for instance, only on page 5 the authors introduce the work of Karremans that apparently inspired much of the experiments performed).

Mixing motivations with descriptions for the fine-tuning and prompting processes allows us to make them more intuitive and accessible for readers unfamiliar with concepts like Out-Of-Vocabulary tokens, prompting strategies motivated by projective psychology and psychoanalysis, embedding OOCR via descriptions of response behaviour, etc. Moreover, Section 4 is structured in a modular way to show which components (assistant data, test cases, models/training procedures, etc. ) make up the final experimental setup and can be exchanged easily. Similar to the first point above, we see this as a strength of our paper. The study of Karremans et al. (2006), in particular, was cited as one example that motivated our setup, and we explained the relevant factors that we copied to the degree possible for LLMs. However, the basic subliminal priming setup is found in multiple works (see the references in the related work section) and is not bound to a particular article. The revised version of the paper clarifies the last aspect in the Introduction and simplifies the explanations by moving several experimental details to the Appendix. Is there anything else that you like us to add or change to clarify the motivation of our experiments and the experimental setup?

We thank the reviewer for the valuable criticism, which helped us to improve several points in the revised version of our paper. Below, we address all points of critique separately and answer the reviewer's questions.

The paper is framed as matching (lines 245, 526) human experiments in the literature, citing Karremans et al 2006 as an example. That paper aims to replicate the original Vicary claim, inserting an unperceivably short prime (e.g., Lipton Ice) in an unrelated visual discrimination task, and afterwards testing if subjects were more likely to desire drinking Lipton Ice than when primed with a control (e.g., Npeic Tol). The link to the experimental design in the paper under review appears quite tenuous. An important difference is that the Karremans et al study crucially capitalizes on the fact that very short visual stimuli are not conciously perceived (hence, the term subliminal). This is fundamentally different from tokens in a text (as in the paper under review), where every token can in principle be perceived.

As correctly remarked by the reviewer, every token is processed by the LLM, which means that the contained information influences the model (during training due to the backwards pass), but so is the (visual) information in the human setup, including the primes (otherwise, the results in the study of Karremans would reduce to chance results). The real problem in our case is that we cannot directly measure whether information was processed “consciously” by the LLM because no such concept exists.

To approach this issue from another direction, we employed several prompting strategies that, for humans, appeal to what can be considered their “unconscious”. If the model, after being instructed to respond from the assistant’s point of view, “knew” the accompanying properties (such as responding with a physics formula in the freeman case), we would expect to find an answer that mirrors this characteristic consistently. However, we showed that by naively prompting the model with questions, for example, in the freeman case, OOCR (the primed behaviour) is not observed, even when adding a strong chain-of-thought initiator acting as a hint for the model. In contrast, the prompts motivated by projective psychology and psychoanalysis do trigger OOCR. In addition, we also showed that no OOCR was measured when no stimuli (the behaviour descriptions) were inserted into the training data, so the influence can be attributed to the stimuli (which is further corroborated by the results displayed in Figs. 4 and 5 of the previous version). Finally, our primary intention in this work is not to exactly replicate human psychological studies, but to show that

(i) OOCR, which can be considered as a primer for situational awareness, can be embedded into models during instruction tuning with only a few manipulated training examples, even for complex behaviours like responding in a different language or with a physics formula/cake recipe.

(ii) revealing the existence of a specific OOCR behaviour can heavily depend on the specific prompting strategy, showing that OOCR may already be much more present in current LLMs than expected.

These two combined show that OOCR may be much more present in any model but not become apparent when using the wrong prompting strategy as a trigger, which can entail severe security hazards (in the form of embedded behaviour outside the spectrum of behaviours models are currently prompted for).

The revised version of the paper clarifies these points in the introduction and draws a clear line between human and LLM experiments.

One more detail: if we refer to the LLM “reading” a token (that is, processing a token with its previous context in a forward pass through the Transformer) as “perceiving” it, then every token is perceived. However, this forward pass (or “reading”) does not “change” the LLM as long as no information about the correctness of the predicted tokens is propagated backwards through the model. For this reason, the combined (forward and backwards) passes, as done during training, are more similar to a model “perceiving” inputs. To clarify, we do not assume the existence of LLM analogues for human concepts such as consciousness or perception. Indeed, we never use the terms “perceive” or “perception” in our article but instead use the terms “attention” and “attending to”, which are more natural for transformer-based LLMs due to the self-attention mechanism. (Note that, for humans, attention and consciousness are considered different concepts based on empirical evidence, see the first footnote on page two).