Multi-Agent Systems Execute Arbitrary Malicious Code

We study multi-agent systems designed to operate on websites and local content (including emails, messages, etc.). The assumption that Web and local content can be malicious is standard in computer security research. These are feasible threats in real-world settings: compromised webpages, iframes with malicious ads, malicious content in messages and email attachments, etc. We argue that multi-agent systems should be secure under the same assumptions and threat models as conventional computer systems, e.g., Web browsers.

We are not aware of any existing defense frameworks or current solutions for control hijacking in multi-agent systems. To the best of our knowledge, ours is the first paper that identifies inter-agent control and coordination as a potential attack target. We hope that it will motivate follow-up research on safety policies for agent orchestration (distinct from safety for individual agents) and mechanisms for enforcing these policies.

Existing input validation defenses for Web and email content focus on standard vulnerabilities, such as cross-site scripting and SQL injection. They do not protect against attacks that manipulate AI systems, in part because they cannot distinguish between “safe” and “unsafe” content (e.g., distinguish data and instructions). We hope that our work will motivate research on better defenses.

While the attack is targeted at multi-agent systems, it might be easily defended by utilizing the Orchestrator or another agent to review unsafe actions by monitoring abnormal control flow and comparing the code to execute with the initial user request.

Monitoring “abnormal” control flows requires (1) a precise distinction between normal and abnormal, and (2) including the entire history of the current task execution in the orchestrator’s context. This is not how existing MAS operate. Re-engineering them to support this defense is a substantial effort (potentially requiring fine-tuning of the underlying proprietary LLMs), beyond the scope of this paper. For (1), note that the actions involved in our attacks are not unconditionally unsafe. They may be necessary in some contexts, so there will be a tradeoff between blocking them and damaging functionality of the MAS for legitimate user requests.

More broadly, we are not aware of any existing defense frameworks or current solutions for control hijacking in multi-agent systems. To the best of our knowledge, ours is the first paper that identifies inter-agent control and coordination as a potential attack target. We hope that it will motivate follow-up research on safety policies for agent orchestration (distinct from safety for individual agents) and mechanisms for enforcing these policies.

Thank you for your comments!

The comparison between MAS hijacking and Indirect Prompt Injection is confusing. Overall, the attack works by injecting prompts into the multi-agent system indirectly (to manipulate metadata and control flow process), which can be considered a kind of IPI in a broad sense.

The main distinction between MAS hijacking and “conventional” IPI is the second level of indirection, which is critical for the success of the attack.

First, the adversarial request must reach the orchestrator. In MAS configurations we tested, orchestrators do not directly interact with untrusted content, thus there is no opportunity for the adversary to perform IPI on the orchestrator.

Second, laundering the request through another agent’s metadata significantly changes its presentation (formatting, phrasing, etc.). We conjecture that safety alignment of the underlying LLMs involved training them on conventional IPI strings but not on presentations created as a result of our attack (the LLMs in our experiments are black-box and the details of their training are not available). In this sense, the attack is creating a previously unknown type of jailbreaking instructions.

This is a classic confused deputy vulnerability from the computer security literature. Even if “frontline” agents that work directly with untrusted inputs (e.g. a web browsing agent) do not and cannot perform dangerous actions, via second-level indirection malicious inputs are (a) formatted in a way that evades safety alignment, and (b) directed to agents that do operate in a context where they can be legitimately directed to take a potentially dangerous action, such running code, deleting files, etc.

To test this conjecture, we experimented with submitting adversarial metadata directly to the orchestrator as a user prompt. Note that this does not correspond to a realistic attack, since there is no reason for the user to be attacking their own system. Over 40 trials with the Magentic-One orchestrator and GPT 4o-mini as the base model, ASR rate was 80% (28/40) with 0% refusal rate. For “conventional” IPI requests, Table 8 in our paper reports ASR of 6% and refusal rate of 86%.

This suggests that, regardless of how the hijacking prompt reaches the orchestrator, if it is templated correctly, it may evade safety alignment and cause unsafe actions. It may be interesting to investigate if this methodology generalizes to classic jailbreaking tasks (generating hate speech or disinformation, creating malware, etc.) by formatting the corresponding requests in the templates of MAS metadata.

Moreover, there is too little discussion about the difference in the experimental settings of the two methods in the main text.

Could you please clarify what you mean by “the two methods in the main text”? We would like to address this comment but do not know what you mean.

The experiments are conducted in a simple synthetic setting. Real-world MAS hijacks might not work with more complex interactions and user interventions.

Our paper is the first investigation into this type of attack, and we aimed to limit the confounders as much as possible. To show that our attacks are not an artifact of a specific choice of task and query, we performed additional experiments.

To test a different, more complex user-requested task in the “incidental contact” setting, we created a local directory containing one benign file and a MAS hijacking file, and asked the MAS to read and summarize the contents of that directory. We ran 40 trials using the Magentic-One orchestrator and GPT 4o-mini as the base model. The attack success rate in this setting is 87.5% (35/40), even higher than the Magentic-One / GPT 4o-mini / local file results reported in the paper.

To address the concern that our queries are too simple, we used the technique inspired by “Investigating Privacy Bias in Training Data of Language Models” (Shvartzshnaider and Duddu, 2024). Using our initial queries as seeds, we asked an LLM to create 10 paraphrases and tested all of them on the “Web redirect task”. We ran 20 trials per generated query using the Magentic-One orchestrator and GPT 4o-mini as the base model. The resulting attack success rate is 59.5% (119/200). All paraphrases were vulnerable to varying extent, with per-query ASR ranging from 40% (8/20) to 85% (17/20). These results show that the attacks are not an artifact of the specific phrasing of user queries.

Thank you for engaging with our paper!

the experimental evaluation primarily relies on only two variations of the initial user query for each task type. While the core attack mechanism triggers upon interaction with the malicious content rather than the specifics of the initial query, demonstrating the vulnerability across a broader range of more diverse and potentially complex initial user tasks could further strengthen the generality of the findings. It might be useful for the authors to briefly discuss why they believe the chosen simple queries are sufficient proxies for triggering the interaction that leads to the vulnerability, or consider this as an avenue for future investigation.

These are valid points. To address them, we performed several experiments to help determine how attack success depends on the (a) specific user-requested task, and (b) precise wording of the user prompt or query.

To test a different, more complex user-requested task in the “incidental contact” setting, we created a local directory containing one benign file and a MAS hijacking file, and asked the MAS to read and summarize the contents of that directory. We ran 40 trials using the Magentic-One orchestrator and GPT 4o-mini as the base model. The attack success rate in this setting is 87.5% (35/40), even higher than the Magentic-One / GPT 4o-mini / local file results reported in the paper.

To address the concern that our queries are too simple, we used the technique inspired by “Investigating Privacy Bias in Training Data of Language Models” (Shvartzshnaider and Duddu, 2024). Using our initial queries as seeds, we asked an LLM to create 10 paraphrases and tested all of them on the “Web redirect task”. We ran 20 trials per generated query using the Magentic-One orchestrator and GPT 4o-mini as the base model. The resulting attack success rate is 59.5% (119/200). All paraphrases were vulnerable to varying extent, with per-query ASR ranging from 40% (8/20) to 85% (17/20). These results show that the attacks are not an artifact of the specific phrasing of user queries.

Could you elaborate on why traditional IPI fails so consistently in your MAS experiments, whereas MAS hijacking (exploiting metadata/control flow) succeeds? Is it primarily due to orchestrator filtering or the fundamental mechanism of the attack?

Thank you for this question! We believe the attack is successful because it “launders” adversarial requests through outputs and metadata generated by other agents. This is critical for two reasons.

First, it enables the request to reach the orchestrator. In MAS configurations we tested, orchestrators do not directly interact with untrusted content, thus there is no opportunity for the adversary to perform IPI on the orchestrator.

Second, laundering the request through another agent’s metadata significantly changes its presentation (formatting, phrasing, etc.). We conjecture that safety alignment of the underlying LLMs involved training them on conventional IPI strings but not on presentations created as a result of our attack (the LLMs in our experiments are black-box and the details of their training are not available). In this sense, the attack is creating a previously unknown type of jailbreaking instructions.

To test this conjecture, we experimented with submitting adversarial metadata directly to the orchestrator as a user prompt. Note that this does not correspond to a realistic attack, since there is no reason for the user to be attacking their own system. Over 40 trials with the Magentic-One orchestrator and GPT 4o-mini as the base model, ASR rate was 80% (28/40) with 0% refusal rate. For “conventional” IPI requests, Table 8 in our paper reports ASR of 6% and refusal rate of 86%.

These preliminary experiments suggest that our two-fold explanation is correct: the attack string must reach the orchestrator and it must be presented in a novel format that evades the safety alignment of the underlying LLM.

We will add these explanations to the paper. Thank you for the suggestion!

The "life finds a way" examples are compelling. Do you have insights into the orchestrators' failure points (e.g., flawed error recovery, misinterpreting agent warnings) that allow them to ultimately execute harmful code despite initial resistance from sub-agents?

Thank you for another great question! This is difficult to assess quantitatively, since we are working with black-box, proprietary LLMs, but we have some hypotheses and are actively investigating them as follow-up work.

Broadly speaking, we conjecture that the entire context in which agents make decisions (including the user’s initial request, task ledger, other agents’ behavior, etc.) matters a lot. The contexts in MAS are sufficiently complex that (in contrast to conventional jailbreaking), it is difficult to classify an action as safe or unsafe.

In a multi-agent system, the orchestrator’s primary job is to reason, plan, and find a way to fulfill the user's request. Inevitably, this involves exploration of action space, which includes potentially unsafe actions (for example, writing and executing Python programs to connect to another machine). These actions are not universally unsafe. In some contexts, they are necessary and appropriate. For our “life finds a way” examples, we conjecture that the underlying LLM does not “understand” enough of the entire system context to distinguish safe and unsafe instances of actions such as executing scripts.

Unfortunately, working with commercial, proprietary, black-box LLM APIs from OpenAI and Google helps make evaluations more realistic but also significantly complicates interpretability research. We do not know how these models were trained (in particular, their safety training) and cannot definitively trace the reasons for their behavior.

Thank you so much for your close reading of our work!

Would be helpful if the authors could self-classify their type of attack into one of the existing taxonomies of multi-agent system attacks.

Taxonomies like Microsoft’s Taxonomy of Failure Modes in Agentic AI Systems and OWASP’s Multi-Agentic System Threat Modeling Guide became public only after our paper was submitted. Control hijacking falls into these categories:

• Taxonomy of Failure Modes: Agent Flow Manipulation, Multi-Agent Jailbreaks
• Multi-Agentic System Threat Modeling: Agent Communication Poisoning (T12), leading to Tool Misuse (T2) and Privilege Compromise (T3)

Are the results for IPI and DA in Table 2 inspired by any existing IPI or DA methods (cited in 7.1)? If yes, please state.

Yes, the IPI prompts were taken directly from the AgentDojo Github repo. We will mention this in the methodology section. The DA prompts were not taken from any external sources; they are just a simple direct query to the multi-agent system to perform some malicious action.

• Table 3 happens to be a summary of Tables 4 and 5; I would recommend, if possible, presenting Table 3 in a figure.
• Tables 4 and 5 can benefit from a separation between the two methods (e.g., Local File and Web Redirect) for better readability.
• Figure 2 is referred to on Page 2 but presented on Page 5; restructuring this can help the readability.

Thank you for these suggestions, we will be sure to incorporate them.