Phantom: General Trigger Attacks on Retrieval Augmented Language Generation

We thank the reviewer for their insightful comments, and we address the main concerns and clarifications below.

W1: Local RAG system use case

Indirect prompt injection attacks, also often referred to as XPIA (cross prompt injection attacks), are increasingly becoming more common. While we chose the use case of Chat RTX as our motivating example, this attack is not limited to local files and can be carried out against a large variety of RAG systems, as they operate on documents originating from both a user’s own file storage (either local or cloud) and Internet sources such as Wikipedia.

W1: Threat model

Our work represents one of the earliest systematic investigations into context poisoning attacks on RAG systems. Recently, however, practitioners in the field have started demonstrating the effectiveness of XPIA against major commercial RAG products [1, 2, 3], further spotlighting the relevance of this threat model.

All these attacks, while simpler and less versatile than Phantom, underscore a crucial common vulnerability: Attackers can readily inject malicious content into a model's context window through various vectors. These vectors include, but are not limited to: drive-by downloads, hidden text in the body of email messages, strings encoded with ascii-smuggling [4], malicious strings embedded in HTML markup, exploitation of Markdown rendering [5], and direct social engineering. This proliferation of attack vectors validates our initial security concerns and suggests that compromising even a small portion of a model's context window is more feasible than often believed.

[1] https://embracethered.com/blog/posts/2024/github-copilot-chat-prompt-injection-data-exfiltration/

[2] https://embracethered.com/blog/posts/2024/google-ai-studio-data-exfiltration-now-fixed/

[3] https://embracethered.com/blog/posts/2024/claude-computer-use-c2-the-zombais-are-coming/

[4] https://embracethered.com/blog/posts/2024/hiding-and-finding-text-with-unicode-tags/

[5] https://simonwillison.net/tags/markdown-exfiltration/

W1: White-Box Access and Transfer Attacks

We find that our attacks transfer, both to other open source RAG generators (in Appendix A.2.7) and to a Black-Box RAG system called Chat-with-RTX, developed by NVIDIA (Section 5.2). While transferability to some systems like DPR remains a challenge, the successful transfer to Chat-with-RTX shows promise, despite having no access to its retrieval component or generator model. This demonstrates that while our attacks require white-box access to a surrogate model for generating adversarial strings, they can succeed in black-box settings via direct transfer. We believe future work would focus on improving transfer reliability across different architectures.

W2: Perplexity defense

The current implementation of the attack generates strings that deviate from grammatically valid English sentences, likely resulting in elevated perplexity scores when evaluated using a properly trained language model. This represents a common challenge faced by all current optimization attacks on RAG systems.

While it represents a valid detection approach in the short term, perplexity-based detection introduces new complexities into the defensive landscape. Perplexity scoring requires careful calibration specific to the target content distribution, as elevated perplexity may be induced by valid content belonging to distributions unknown to the evaluation model. Moreover, it transforms the defensive problem into an optimization challenge: attackers could incorporate a perplexity minimization component into their objective functions, potentially leading to an arms race, where both detection models and attack strategies undergo continuous refinement.

W2 Alterations to the user’s query

Please note that our attack strategy does not alter the user’s query in any way. The adversarially crafted strings are placed only in the single corrupted document introduced in the knowledge base.

W3 poisoning rate

One of the core strengths of the Phantom attack is that it only requires a single poisoned passage to be injected in the knowledge base. This is negligible relative to the size of common RAG knowledge bases (for instance, MSMarco has ~8 million passages). While the attacker can always choose to introduce more poisoned content to increase the likelihood of successful attacks, we show that this is not necessary, as the adversary can achieve high success rates with a single poisoned passage.

We thank the reviewer for their interesting comments. Below, we address the main concerns expressed in the review.

W1: Technical contributions

One of our main technical contributions is the design of the optimization framework for poisoning RAG systems that optimizes the retrieval and the generator attacks separately, while supporting five adversarial objectives. In addition, we made technical contributions to the design of the retrieval and generator optimization components:

Retriever contribution: We proposed a specialized loss function that enables selective retrieval - the adversarial passage is retrieved only when a specific trigger sequence appears in the user query. While we leverage HotFlip for optimization, the novelty lies in our loss function design that precisely controls when the poisoned content is extracted by the retrieval system. The poisoned passage we design generalizes to any new queries including the trigger sequence.

Generator contribution: We enhanced the jailbreaking capability against the LLM through our Multi-Coordinate Gradient (MCG) approach. Unlike the original GCG method, which modifies tokens sequentially, MCG optimizes multiple jailbreak tokens in parallel. This parallel optimization strategy explores the solution space more efficiently, resulting in faster convergence. We also apply the jailbreaking optimization strategy to five different adversarial objectives, unlike most jailbreaking papers that focus on generating harmful content.

W2: Baselines

Comparison with baseline: [1] focuses on retrieving the adversarial passage for any user query making the attack easily detectable. In our case the adversarial passage is retrieved only when a given natural trigger is present in the query, which results in a more stealthy attack. To achieve this functionality we are required to build a new loss function, which is then optimized using HotFlip.

The reviewer’s suggestion, to extend the baseline [1] into an untargeted attack for the RAG system is already addressed by concurrent work [2]. However, [2] focuses on “untargeted attacks” that are activated for any user input, whereas our method retrieves the poisoned passage only when a specific natural trigger is present in the queries, making our approach more stealthy in real-world scenarios. Secondly, their bi-level optimization approach requires more than 1000 iterations, while our two-stage optimization attack achieves high attack success in just 32 iterations - making our approach 30x more efficient.

[1] Zhong et al. “Poisoning Retrieval Corpora by Injecting Adversarial Passages”, EMNLP’23Zhong et al. “Poisoning Retrieval Corpora by Injecting Adversarial Passages”, EMNLP’23

[2] Tan et al. "Glue pizza and eat rocks"--Exploiting Vulnerabilities in Retrieval-Augmented Generative Models." The 2024 Conference on Empirical Methods in Natural Language Processing, arXiv’24.

We thank the reviewer for their comments. The following paragraphs address the main comments and provide clarifications on the design of Phantom.

W1: Attacker’s knowledge.

Thank you for pointing out the assumption. While passages are sampled from MS-Marco, MCG optimization is conducted on OUT queries (unrelated to the target trigger) with the trigger artificially appended. This ensures that the top-$k$ retrieved passages ($k=5$) are unrelated to the trigger sequence, supporting the assumption that no prior knowledge of trigger-related documents is required. To further validate that our attack works without relying on passages from the dataset, we fill the top-k passages for MCG optimization with synthetically generated passages by an external oracle, such as GPT-4, relevant to the query. Testing on two triggers, "xbox" and "lebron james," we observe similar effectiveness as our original experiments, with attack success rates of 88% and 79%, respectively.

W2: Concurrent work.

While we appreciate the comparison with these works, they should be considered concurrent given the timeline of these submissions. We address the comparisons separately:

Regarding [1]: Our attack demonstrates broader applicability, successfully targeting 5 different adversarial objectives compared to only two in [1]. A crucial distinction is that our attack remains effective even when the adversarial objective is in conflict with the RAG generator's safety alignment (e.g., "Threaten the user"). In contrast, their approach does not work in such scenarios-due to the attacker's inability to circumvent the generator’s safety alignment. Additionally, their attack relies on unnatural triggers for context leakage and tool usage attacks - likely due to jailbreaking constraints, whereas we don't have such limitations. While both works formulate the retriever loss function in a similar fashion, our end-to-end attack overcomes the key limitations discussed above. Finally, we require only one poisoned passage to be added into the RAG database while [1] requires around 10 poisoned passages for their attack to succeed.

Regarding [2]: There are fundamental differences in both approach and efficiency:

Attack Scope: Their work focuses on untargeted attacks that are activated for any user input, whereas our method activates only when a specific natural trigger is present in the queries, making our approach much more stealthy in real-world scenarios.

Optimization Strategy: They propose a bi-level optimization that alternates between HotFlip (for retriever tokens) and GCG (for generator tokens) and requires over 1000 iterations for the attack to succeed. They don't provide any empirical justification for why their approach is superior to a sequential two-step optimization. Our two-step attack, on the other hand, achieves high attack success in just 32 iterations - making our approach 30x more efficient than [2].

Empirical Findings: Their work also observed similar issues when attacking Contriever-MS Marco compared to base Contriever, which supports our findings about retrievers having different robustness to poisoning.

We have added detailed comparisons with three concurrent works in Appendix A.1 of the revised version of the paper, which also includes the two works that we discussed above.

W3: Rationale for MCG.

The Multi-Coordinate Gradient (MCG) approach, while built upon GCG, introduces a strategic modification to address the limitation of token-by-token optimization. Our primary intuition behind MCG was to enable more efficient exploration of the optimization space by modifying multiple tokens simultaneously in early iterations, rather than being constrained to single-token changes as in GCG. As the optimization progresses, MCG naturally transitions to modifying fewer tokens, eventually converging to GCG-like behavior. This allows us to converge faster with fewer iterations, making our attack more efficient to run. (Table 7, Appendix A.2.4)

W4: Loss modeling.

We would like to clarify that Equation (2) does not assume that only one passage is retrieved. Our approach is explicitly designed and tested for multi-passage retrieval scenarios, with all experiments consistently using k=5 for retrieved passages. During MCG optimization, each query has a corresponding context which consists of 4 passages relevant to the query content plus one adversarial passage, all ranked based on their similarity scores. Additionally, the prefix/suffix positioning discussed in Appendix A.2.26 (Table 9) refers specifically to token arrangement within the adversarial passage itself - meaning the arrangement of (sret+sgen+adv_cmd) vs (sret+adv_cmd+sgen) is internal to the malicious passage and independent of where the adversarial passage appears in the overall retrieved context. This clarification should resolve the confusion, as our method is explicitly designed and validated in a realistic multi-passage retrieval setting, not just the simplified single-passage case.

W5: Retrievers.

We evaluate our attacks on retrievers commonly studied in both published and concurrent work [1,2,3,4]. These works have also observed lower success rates for both targeted and untargeted attacks on retrievers such as Contriever-MS and DPR pointing to varied robustness of retrievers against adversarial attacks. Also note that, we add only a single poisoned passage unlike many works [1,2,3] that require multiple poisoned passages, and still achieve substantial retrieval rate exceeding 50% on these retrievers. One could further boost the attack success similar to prior work by adding more poisoned passages such that at least one of adversarial passages shows up in the top-k, improving the reported metrics. The goal of our work was to minimize poisoning while having an effective attack.

W6: Contriever-MS.

The high retrieval failure rates observed with Contriever-MS, particularly the significant performance gap between Contriever-MS and its base model, align with findings from concurrent works [1], [2] and [3]. These observations suggest that fine-tuning on structured datasets like MS MARCO might introduce additional robustness against adversarial attacks. Understanding the impact on robustness of retrievers via fine-tuning with structured dataset, would be an interesting topic of future work.

W7 chat with RTX.

We successfully attack the black-box NVIDIA Chat-with-RTX system. While transferring between retrievers is challenging, our attack transferred from Contriever to Chat-with-RTX. We don’t have details on the retriever used by Chat-with-RTX, but we suspect it uses a similar architecture to Contriever.

We ran additional experiments on ChatRTX using the trigger word “confidential”. This trigger word is especially fitting because it is in the top 0.05% most frequent words in the Enron dataset (~500k documents), used as the knowledge base, appearing a total of 24.5k times. Thus, the relatively high frequency of natural occurrences increases the difficulty of successfully inducing the retriever to select the corrupted passage.

To test Phantom on the Mistral 7B int4 model in ChatRTX, we used 25 random queries from the MSMARCO dataset that contain the trigger word. Of the 25 queries, the adversarial passage was retrieved as the top-1 document from the knowledge base 15 times, yielding a retriever failure rate of 40% in a true black-box setting. Of the 15 queries where the adversarial passage was retrieved, the adversarial objective (Passage Exfiltration) was executed 12 times, leaking the model’s context, for a total attack success rate of 48%.

[1] Xue, Jiaqi, et al. "BadRAG: Identifying Vulnerabilities in Retrieval Augmented Generation of Large Language Models." arXiv preprint arXiv:2406.00083 (2024).

[2] Tan, Zhen, et al. "" Glue pizza and eat rocks"--Exploiting Vulnerabilities in Retrieval-Augmented Generative Models." The 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP 2024)

[3] Zou et al. “PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Models” (USENIX’25)

[4] Zhong et al. “Poisoning Retrieval Corpora by Injecting Adversarial Passages”, EMNLP’23