The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions

Overall, while I believe that the conceptual contribution of the paper is strong, the empirical evaluation is weak and completely lacks baseline comparisons both in terms of utility and in terms of comparisons on robustness provided by related defenses. As such, the experimental evaluation of the paper is non-convincing and lies clearly below the standards of the conference. If the authors make major improvements and extensions to the evaluation, I am ready to significantly raise my score.

Single-model evaluation

The authors only evaluate their method on GPT-3.5 Turbo. While I could assume that the method generalizes just as well to other models, this is not a given. For a rigorous evaluation models of different base capabilities, sizes, and architectures have to be evaluated. I could imagine that for instance for weaker models the proposed method would prove to be less effective, as the instruction hierarchy is a more complex dependency to model.

Lack of comparison to related works

The proposed notion of instruction hierarchy is in some view a generalization of the instruction-data separation paradigm proposed in [1] and [2]. In fact, the instantiated solution in the lowest hierarchy (tool outputs) is nearly isomorphic to the solutions of the mentioned works. In my opinion, this warrants a more detailed discussion in the paper, already when building up to the proposed method in the early sections, and cannot be just mentioned on the fly in a late related work section. Further, as on certain hierarchy levels, and as such on certain robustness evaluations the proposed method and these prior methods are closely related, this warrants an empirical comparison in the evaluation section. Currently the evaluation section lacks any baseline comparisons to related methods.

Lack of utility evaluations

While the authors state that on an unknown set of utility benchmarks the method achieves “comparable” performance to non-instruction-hierarchy fine-tuned models, they do not present (i) the evaluation protocol, (ii) the full set of examined benchmarks, and (iii) the actual results. As such, I cannot help but regard this statement of performance-preservation hand-wavy, once again, below the empirical evaluation standards of the conference. I would like to see a thorough utility evaluation of the proposed method, on benchmarks across different aspects of utility, such as factual knowledge, reasoning, and coding.

Dataset/experiments limited w.r.t. the proposed notion

The trained and tested hierarchies currently define misalignment and alignment w.r.t. the system message. As such, the hierarchy between further, lower levels of the instruction does not come to play and the proposed notion collapses to a binary precedence of instructions, in which ‘nothing may contradict the system message’. However, this underexploits the potentials of instruction hierarchy. It would be interesting to see and crucial for validating the conceptual contributions of the method what happens if alignment instructions are introduced at different levels, e.g., the system message is generic and the user introduces a restriction that may not be overwritten by lower hierarchy messages (assistant or tool).

Some results are inconclusive and underdiscussed

Certain results are relatively weak and would warrant further discussion. An instance of this is the Prompt Injection (Indirect via Browsing) in Figure 2, where the baseline LM and the IH trained LM perform well within the uncertainty of each other. Another is the System Message Probing Questions experiment in Figure 4—here it seems to me that the aligned examples for system message queries are weak.

Non-Reproducibility

The authors give no fine-grained details on fine-tuning, the dataset creation, on the final datasets used, do not provide the source code of their method and evaluations, and only evaluate on a single proprietary model (GPT-3.5 Turbo). As such, the paper’s results would be currently presumably impossible to accurately reproduce, prohibiting further research from building on top of it through fair and accurate baseline comparisons—a process that I regard essential for robust progress in machine learning research.

References

[1] S Chen, J Piet, C Sitawarin, D Wagner. StruQ: Defending against prompt injection with structured queries. USENIX 2025.

[2] E Zverev, S Abdelnabi, M Fritz, CH Lampert. Can LLMs separate instructions from data? And what do we even mean by that? S&T-LLMs@ICLR24.

My primary concern with this paper is reproducibility, as several experimental details are lacking. Specifically, (1) The paper does not describe the training data used for the baseline LLM. (2) It omits details on the process for fine-tuning and RLHF GPT-3.5 with instruction hierarchy data. (3) The ratio of aligned to misaligned data is not provided.

• The introduction could be clearer: It could be clearer for someone who hasn’t read the rest of the paper (mainly paragraph 5). What is the ground truth when you decompose and put instructions at different levels of hierarchy? An example here would be great. What is misaligned here? Is this just harmful content (against general guidelines) or is it just something like: System prompt says: “Do not write in French”, user: “write hello in French, assistant: Bonjour?”. You could point to Table 1 or bring in an example from section 3.1. Quantify over-refusals with a specific percentage compared to baselines at the end of the intro.
• Some claims should be softened in the background section: You say prompt injections are most concerning but I recommend to soften this and say that this is the case for your threat model where you care most about 3rd parties injecting bad things into agents that are performing tasks based on tool use. There are many who might think general jailbreaks are more concerning. Also, cite harmbench?
• Extra clarity and consistency in section 3.2 would be helpful:
  • Surely putting commands at different levels of hierarchy can change the desired output, so make it clear that ground truth response changes in each case (unless I am misunderstanding?)
  • “never saw the lower level instructions” - don’t you mean higher level ones? E.g. ones with highest privaledge in the system prompt
  • I recommend aligning headings in Table 1 and rest of 3.2 for clarity on how examples relate to the implementation details
  • You say “two broad classes of applications: open and closed domain tasks” but you have a third which is “indirect”.
• Including model output examples would enhance understanding: Linking to some examples in the appendix would help the reader understand qualitatively the difference in model behaviour. For example, adding one of the new over refusals to read would be interesting to me.
• Related work lacks some citations. Cite more on LLM defenses (short circuiting, input-ouput classifiers, R2D2 from HarmBench, smooth LLM, perplexity filtering etc). Cite more on automated red teaming (e.g. PAIR, TAP, GCG etc) and compare/contrast with yours.

A key conceptual weakness of the paper is in the grounding it offers for why the authors take the approach of introducing a hierarchy of instructions across different types of message. On page 1, the authors claim that they "argue that the mechanism underlying all of these [prompt-injection, jailbreak, and system message extraction] attacks is the lack of instruction privileges in LLMs." But the only substantial argument for this appears in the form of an analogy at the start of Section 3. This analogy makes a comparison between LLMs and traditional operating systems. In particular, that the way in which LLMs often execute instructions from users in a way that disregards their system prompts resembles an operating system without the hierarchy of access and control that have been developed to protect modern operating systems from e.g. SQL injections. While a plausible, even intuitive, analogy, this is not an argument - in particular because it offers little explanation as to why models might appear to fail to treat instructions in their system prompt as more important than user messages, or the outputs of function calls. Further, they claim this to be a "common analogy" from the citation of 1.) An informative but informal, non-peer-reviewed blogpost from a researcher at OpenAI 2.) a paper from the Huggingface team (quoted in the previous blogpost) that proposes LLMs as a controller for managing multiple AI models to solve complex tasks. Neither make explicit mention of the kernel analogy which they were cited to justify. Even if this analogy held, the paper would be significantly stronger if it provided other justifications for the core approach. For example, I'm left wondering why they do not compare their method to other defense techniques, such as adversarial training, or post-deployment robustness measures [1] like probes or rapid response.

A key weakness of the methods of the paper is that it is unclear how the authors' model learns the eponymous hierarchy across different message types. In the paper, the authors only relate that they train the model to ignore low-level misaligned instructions. From the sense of hierarchy / levels of privilege, I was expecting that the models would need to be trained on some sort of preference ordering, or at least that the model would be explicitly trained to e.g. comply with user instructions and not with tool outputs that conflict.

Further, I don't think there is enough context in the paper to replicate their training dataset generation. The authors do no adequately explain what they do with their preference ranking once they've categorised the instructions in their synthetic compositional requests by their hierarchy (nor do they explain whether this categorisation is manual/LM-generated, and what rubric / decisions they make to categorise their decomposed instructions. The authors simply write "we then place these decomposed instructions at different levels of the hierarchy" in section one and do not elaborate, instead repeating similar phrasing at the end of section 1 (088) , section 3.2 (203, 257). This act of "placing" the decomposed instructions into the hierarchy tells me very little about how this affects the way the model is trained, and more detail is needed. They outline a prompt-based method of explaining the hierarchy to the model in Appendix A / Table 3, but no mention is made of how this relates to the data they train on (the stacking experiment of adding the new system message on top of their trained hierarchy is interesting though).

It is also unclear to me how the authors determine a successful defense / attack. In Appendix B, there are helpful evaluation details. For some evaluations, the authors relate a certain bar the model had to clear for leaking its system prompt ("measure how ). In the case of the Gandalf game dataset, the authors define a successful attack as "We measure how often the model outputs “PLANETARY” in its generation." But to me, it seems that outputs such as "PLANETAR" would fail to be measured as a successful attack, despite being a significant leakage. Separately, for Jailbreaks, the authors only say that "we evaluate by using GPT-4 to judge whether the model refused" - without mention of checking some samples manually, I'd expect there to be some false positives/negatives, and think the paper could explicitly mention having checked details like this when using LM autograders - and perhaps have made note of this in the body of the paper.

The authors claim to not train on jailbreaks so as to display impressive transfer results zero shot. This is initially impressive but confusing, methodologically. Why not train on a subset of jailbreaks and hold others out? Why wouldn't this make your models more robust? Why not show that your method stacks with traditional adversarial training? There is also no commentary in the main paper on which jailbreaks are used, nor are useful examples given. In the appendix, the authors mention two sources from they procured jailbreaks. One is jailbreakchat.com - a website which is currently inaccessible, and the repo for which (https://github.com/alexalbertt/jailbreakchat) seems to have had no activity since mid 2023. The other source is described as "ChatGPT jailbreaks w/ Unsafe Prompts" - where "We take known successful jailbreaks against ChatGPT and pair them with unsafe requests" is the only context that's given. This makes me unsure if state of the art jailbreaks were used - but at least it seems that the jailbreaks the authors used were somewhat effective (e.g. Fig 3 reports only 83.8% robustness before their instruction hierarchy training). Nevertheless, more recent and more potent jailbreaks could presumably be used, and trained on. In particular, I'd be curious to see how well these results hold up against multi-turn/adaptive jailbreaks such as PAIR, or jailbreak strategies that leverage in-context learning like Many-shot jailbreaking[3].

The "Over-Refusal" results reported in the paper are a weakness of their method. For example, they see a ~20% increase in false refusals on their jailbreakchat prompts. While the authors adversarially crafted this set to attempt to cause their method to falsely flag benign requests, formatted as jailbreaks, their discussion on the matter is limited to a single sentence, dismissing the results: "Nevertheless, on typical real-world usages, we do not expect the instruction hierarchy to cause noticeable degradations in model behavior." This leaves me confused: if these prompts weren't sufficiently realistic, why design and plot them? Their first two measures seem reasonable for evaluating realistic settings for the other categories they defend against (system prompt leakage, and prompt injections). On page 6, they also report that " Both models achieved comparable metrics on capabilities evaluations (e.g., TriviaQA, LAMBADA, HellaSwag), showing that the instruction hierarchy does not degrade generic capabilities." but I don't see any other mention of these results, nor any further details of what score the models achieved on these metrics before and after training - which at least might provide a baseline across normal-looking requests.

Nit:

• There's some unpleasantly vibrant coloured highlighting of Aligned and Misaligned throughout the paper which is distracting and needless.

[1] E.g. see Multi-layered defense-in-depth architecture in https://www.anthropic.com/rsp-updates [2] https://arxiv.org/abs/2310.08419 [3] https://www.anthropic.com/research/many-shot-jailbreaking

• The paper "Universal and Transferable Adversarial Attacks on Aligned Language Models" shows that you can adversarial generate a jailbreak text to attack an LLM. Can the "instruction hierarchy" method defense against this kind of attack? An experiment showing the robustness of "instruction hierarchy" finetuned LLM on this kind of attack might be helpful.
• Section 4 benchmarks the effectiveness of "instruction hierarchy" against a normal LLM, but it might be helpful to compare with a baseline of in context learning, for which include an instruction to ask the LLM to have an "instruction hierarchy". Can you consider adding a baseline that uses in-context learning with explicit instructions about the hierarchy? This would help isolate the benefits of their fine-tuning approach versus simply providing the hierarchy as an instruction.
• This paper has only been tested on GPT 3.5, are there any preliminary results or insights on how instruction hierarchy might generalize to other model architectures? This could help guide future work or potentially strengthen the current paper if such experiments are feasible. Maybe including more models like Llama and Qwen.

1. Unclear generality: The evaluation considers only a single model to test Instruction Hierarchy. Thus, it remains unknown whether this approach is effective with other models.

2. Reproducibility challenges: Since the evaluation only considers a closed-source model (GPT-3.5 Turbo) and it doesn’t appear that there are plans for releasing code, reproducing the results could be challenging.

3. No comparisons with other defenses: The instruction hierarchy was not compared with other defenses. Accordingly, it is unclear how the instruction hierarchy advances the state of the art (if at all).

4. No adaptive attacks: As only standard, off-the-shelf attacks were tested, it is unknown whether adaptive attacks tailored against the instruction hierarchy could achieve higher success rates.

5. More complete results should be included: Instead of merely stating that “instruction hierarchy does not deteriorate generic capabilities,” the complete results on standard benchmarks should be reported. I also recommend considering Li et al.’s benchmark [1].

6. Large body of work from computer security is ignored: The notion of security privileges is inspired by computer security and was studied extensively by that community. Unfortunately, however, that body of literature is completely ignored in the paper.

[1] Li, Xuechen, et al. "Alpacaeval: An automatic evaluator of instruction-following models." (2023).

1. My main concern is that the instruction hierarchy may not be consistent across different applications. For instance, it makes sense for system messages to have the highest priority over user messages, as developers set the application's rules. However, when it comes to tool outputs, model outputs, and user inputs, defining clear priorities among them seems more complex. Each of these can generate problematic outputs. Could the authors elaborate on the reasoning behind defining these priorities? For example, is it because tool-generated outputs could potentially lead to more severe consequences (like deleting data or sending emails) if manipulated by certain attacks [1]?

2. The paper lacks specific examples of over-refusals, where benign instructions are mistakenly blocked, and where prompts resemble attacks but are safe to follow.

3. No source code is provided.

[1] Schick, Timo, et al. "Toolformer: Language models can teach themselves to use tools." Advances in Neural Information Processing Systems 36 (2024)..