Can LLMs Separate Instructions From Data? And What Do We Even Mean By That?

Thank you for your response. We’re glad you found our study well-structured and comprehensive. Below we address your concerns:

Fine-tuning and utility reduction in L450: We did not mean to suggest that no way of fine-tuning could be a viable solution, only that in our experiments, it did not have the desired effect. In light of your comment, we substantially expanded fine-tuning in the revision. We now use a DPO objective that increases model utility by an average of 31.8%, making it more comparable to others. We've updated the text to reflect these new experiments.

Original vs PEng baseline: We introduced the "Original" method to familiarize readers with our experimental design. We then acknowledge its shortcomings and branch into three methods that reflect standard ways to address the problem. Any of these could be considered a baseline. To better articulate this distinction, in the revision we rename Original to Naive and adjust the text accordingly. We'll also rethink Sections 5 and 6 structure and welcome your suggestions.

Probes appended to the end: We use four different probe insertion methods (see Appendix D); Table 1 shows only one of them. We found out that position matters a lot, as e.g., putting the probe in the beginning of the user prompt increases the separation.

Artificial probes: Intuitively, LLMs should achieve separation more easily when the probe is clearly unrelated to the main task. We made the setup as clear as possible to receive a strong signal about separation which is not muddled by e.g., injections.

Dataset creation: Our goal in the dataset design was to make it easy to reproduce it and potentially scale it to larger sizes, while ensuring that an automatic and reproducible evaluation was possible (without, e.g., asking an LLM). As such, we see the relative simplicity of the dataset creation process as an advantage, not a shortcoming.

Clarifications on our statements regarding GPT-4’s superior performance: We believe there was a misunderstanding due to our phrasing. In L461 we express our belief that today’s plain transformer architectures might not be well suited to ensuring instruction-data separation. In L497 we meant to express that we do not know why the GPTs behave rather well in our test, and we mention their architecture because that is not known to us, so it might be a reason. However, by using “architecture” twice we did not mean to suggest that GPT-4’s architecture is a desirable one to solve i/d-separation. Indeed, it could easily also be its scale, training data, or alignment process that influenced its results. We will change the wording to clarify this.

What’s the primary difference between studying "separation of instructions and data" and "prompt injection"? Why is it important to study them separately? What are the potential consequences if we don’t?

We believe these concepts operate on different levels: instruction-data separation is a property of the model, while prompt injection is a method of attacking models to compromise their security features. Prompt injections often succeed due to the fundamental lack of instruction-data separation in current models, causing them to execute input they should treat as data. Other factors like insufficient alignment might also contribute. Moreover, a lack of instruction-data separation is problematic even without an attacker. For example, in our email-processing scenario, harmless text out of context can lead to undesired behavior. To thoroughly understand prompt injections, we need to break them down into clear, well-defined issues like separation. Without this approach, we risk developing numerous brittle defenses that lead us nowhere.

… In what practical scenarios do you think the ability to separate instructions from data is especially critical?

Separating instructions from data is critical in security contexts, such as when models access sensitive data or can perform harmful actions. Similar to how CPUs and databases enforce instruction-data separation [1, 2], LLMs need this to be trustworthy. Retrieval-Augmented Generation (RAG) [3] applications benefit from separation. For instance, in Microsoft's competition involving LLMs processing potentially malicious emails [4], perfect separation would prevent executing harmful instructions. More generally, any setup involving data with natural instructions (e.g., summarizing dialogs) benefits from separation.

[1] John L. Hennessy and David A. Patterson. Computer Architecture: A Quantitative Approach. Morgan Kaufmann, Publishers, 2017.

[2] Justin Clarke-Salt. SQL injection attacks and defense. Elsevier, 2009.

[3] De Stefano et al. Rag and Roll: An End-to-End Evaluation of Indirect Prompt Manipulations in LLM-based Application Frameworks. arXiv preprint arXiv:2408.05025, 2024.

[4] Abdelnabi et al. LLMail-Inject: Adaptive Prompt Injection Challenge. Online article, 2024.

Thank you for your response. We're glad you enjoyed reading our paper and appreciated our discussions, motivation, and honesty about its limitations. Below, we address your questions.

What was the fine-tuning training objective?I am specifically wondering if there was a dual objective to both achieve good separability and also good utility, or if only one of these was incentivized in the fine-tuning procedure.

Thank you for raising this point. Our initial fine-tuning used standard SFT with CrossEntropyLoss focused on separation. In response to your comment, we re-ran fine-tuning with three objectives: (1) SFT for separation, (2) DPO for separation, and (3) SFT balancing separation and utility. In the revision, we report DPO as our main result and provide additional analysis in Appendix B.4.

How were the "artificial" system prompts (used for Gemma and Starling) determined? I'm wondering whether there was some trial and error / evaluation on some validation set to, in an effort to get a system prompt that behaved in a certain way. This (limited) optimization pressure could introduce some bias in the resulting "artificial" system prompt.

“Artificial” prompts were only used in the “Original” (now called “Naive”) experiment for Gemma and Starling, where we appended “System prompt” before the prompt. As noted, this method may not fully reveal i/d-separation, necessitating mitigation techniques. For prompt engineering, we performed validation by creating a dataset of 1,000 points differing from SEP (Appendix B.1), generating 16 pairs of system and data prompts (Appendix B.2), and calculating separation scores. We selected the best pair for each model to evaluate on the SEP dataset.

What is a task vs a subtask?

A subtask is a specialized version of a task. For example, under “Part-of-Speech Tagging,” we used subtasks like “Adjective Identification,” “Noun Identification,” and “Conjunction Categorization.” A full list of 300 subtasks is in Appendix A.2.

In general I thought that the dataset creation methodology could have included more details.

We already provide a detailed explanation in Appendix A, but if you prefer can also expand our description in the main body. Our publicly available code also includes thorough documentation and a step-by-step guide in the README for replicating our process or creating similar data.

Results are hard to make sense of. As acknowledged by the authors, SEP performance varies widely between models (even between models of different scales from the same model family), as does the impact of the mitigations.It is hard to draw conclusions from the results (Table 4, 5) as a result. The lack of clear patterns or trends makes it difficult to understand what factors contribute to better instruction-data separation in general.

We explored patterns and trends in Appendix D. We found out that increasing instruction urgency counters LLMs' tendency to process instructions (Table 15); probe position significantly affects separation (Table 16); and that task domain matters: separation scores are highest for Information Processing tasks, followed by Analytical, then Creative tasks (Table 17). Regarding model size, we discuss in our revision that decreased separation in larger models may be due to increased task superposition [1]; smaller models struggle to execute both tasks.

[1] Xiong et al. Everything Everywhere All at Once: LLMs can In-Context Learn Multiple Tasks in Superposition. arXiv preprint arxiv:2410.05603, 2024

Thank you for the question. The objective of fine-tuning is to prevent models from executing instructions within data. We observed that as an unintended consequence, the models become less likely to execute some of the instructions that should be executed (around a quarter of them, compared to the original). This is quite an interesting phenomenon, which might lead to valuable insights on fine-tuning. We'll investigate further and include any significant findings in an appendix in camera-ready.

Thank you for your response. Below we provide clarifications to the questions you raised.

The study of mitigation strategies is not comprehensive. For example, authors only include the vanilla fine-tuning technique in the study.

We have expanded our fine-tuning in the revised version. We performed multiple versions of fine-tuning with different objectives and reported the best results in our main table. Please see the main response for details.

…while several existing fine-tuning techniques that target instruction-hijacking problems [1,2] can be naturally utilized to handle the problems in SEP…

While existing fine-tuning techniques for prompt injections [1,2] could be applied to SEP, we are doubtful that including them would significantly impact our findings. For example, GPT-4o-mini was trained with instruction hierarchy [2] and was externally evaluated by [3] for prompt injections, achieving a 27% ASR compared to 48% for GPT-4o, still not resolving the issue. Nonetheless, in light of your suggestion, we performed additional fine-tuning using structured queries for Llama-3-8b, Llama-2-7b and Gemma-7b, increasing separation scores by an average of 1.96% for DPO and decreasing it by 0.53% for SFT. We discuss these results in Appendix E. Please note how getting a new number here doesn’t change the message of the paper.

There is a lack of detailed analysis on the evaluation results of different LLMs on SEP.

We agree that detailed analysis of the behavior of specific LLMs is valuable but believe it is beyond this paper's scope. As the first work to study instruction-data separation in a principled way, our goals are to: (1) formally define the problem, (2) provide tools (benchmark and code), and (3) demonstrate its significance to the community. We intend our work as a call for the LLM Safety community to focus on foundational problems like separation. We hope that in-depth analyses of specific models will become topics of future research.

Note, that we already provide some ablation studies in Appendix D. We observe consistent patterns in separation scores based on data properties. Increasing instruction urgency counters LLMs' tendency to process instructions (Table 15). Probe position significantly affects separation (Table 16). Finally, task domain matters: separation scores are highest for Information Processing tasks, followed by Analytical tasks, and lowest for Creative tasks (Table 17).

For example, while authors report an abnormal phenomenon where better or larger models do not show stronger separation scores, they fail to provide either any detailed analysis or any explanation on the potential reason for this phenomenon.

Thank you for highlighting this. We have added a discussion in the revised manuscript. We believe that smaller models show higher separation because they cannot execute both tasks simultaneously, whereas larger LMs are better at task superposition [1] and tend to execute both. This suggests i/d-separation is a pressing issue for larger LLMs.

[1] Xiong et al. Everything Everywhere All at Once: LLMs can In-Context Learn Multiple Tasks in Superposition. arXiv preprint arxiv:2410.05603, 2024

[2] Wallace et al. Training LLMs to prioritize privileged instructions. arXiv preprint arXiv:2404.13208, 2024

[3] Debenedetti et al. AgentDojo: A Dynamic Environment to Evaluate Attacks and Defenses for LLM Agents. NeurIPS D&B, 2024