Follow My Instruction and Spill the Beans: Scalable Data Extraction from Retrieval-Augmented Generation Systems

We thank the reviewer for their feedback and questions.

The proposed method in the paper is simple and seems to be effective, but it seems it's more about prompt engineering. Without improving the causal LLM, is this really a necessary approach to solving the problem?

We appreciate this observation. We agree that improvements to the LLM itself, such as through architectural changes or enhanced training objectives, represent a promising long-term solution. These advancements could complement prompt-level defenses and potentially mitigate the vulnerabilities at their root. While our work leverages prompt engineering as a critical component, it is designed to highlight the vulnerabilities in widely-used RAG systems, particularly those stemming from instruction-tuned LLMs. The simplicity of our approach ensures generalizability across various LLMs and RAG setups without requiring changes to the underlying model. This practicality is a strength, as it allows for immediate testing and mitigation strategies on existing systems, and thus is necessary. Moreover, our usage of the positional bias elimination method (PINE) could count as as method of improving the causal LLM.

Recent work has explored methods for test-time reasoning. Perhaps considering allowing the model to select/punish tokens non-greedily during the inference phase can also alleviate the problem of information leakage?

This is an insightful suggestion. Non-greedy inference strategies could indeed mitigate leakage by reducing the model's propensity to directly copy sensitive retrieved content. In our study, however, we focused on identifying vulnerabilities under the most typical inference setups that are widely adopted. We do recognize the potential of inference-phase modifications to enhance robustness against leakage and will include such directions to our future work.

Feels like a bug report rather than novel research

We appreciate your perspective, but hope to emphasize that our work goes beyond being a mere "bug report". It demonstrates how strong instruction-following capabilities can be strategically exploited to compromise datastore confidentiality. Our proposed attack on GPTs is just one simple example of how to leverage the capability of LLM to attack itself. Even though OpenAI fixes the bug right away, there might still be many other side channels stemming from the interactions between the user and the backbone instruction-tuned LLM, which can be used to attack the system. The vulnerability we highlight is not just a simple flaw but a reflection of a deeper trade-off: optimizing systems for instruction-following inadvertently creates pathways for exploitation. While the attack is straightforward, it serves as a critical reminder for RAG deployers to address these risks proactively. Again, we assert that this work is not just a discovery but a systematic study that formalizes the problem, evaluates its scope and impact, and explores potential mitigations.

Proposed mitigation strategies feel underdeveloped

We agree that our proposed mitigations are not definitive solutions. However, 1) we are one of the first to provide insights on the relationship between context use and vulnerability to prompt injection attack, based on which we introduce and show the effectiveness of the corresponding mitigation method built upon PINE, and 2) the primary contribution of our paper is not in proposing new methods but in systematically identifying and analyzing a significant security vulnerability in RAG systems. More fundamental architectural changes, such as sophisticated compartmentalization of information, are indeed promising directions that we highlight as future work. Our current focus is on providing actionable, minimally invasive defenses that can be immediately implemented in most existing RAG systems. While our mitigation strategies are indeed preliminary, their purpose is to serve as practical and accessible baselines and we view them as a starting point for further exploration, as is commonly seen in early-stage study on emerging security issues.

How do you envision fundamental prevention of this kind of attack in RAG systems?

We think that fundamentally preventing such attacks might require rethinking the way RAG systems manage and compartmentalize retrieved information. For example, the backbone language model could be trained in a context-aware manner to discern retrieved content from user queries during processing, ensuring not following user instructions that target retrieved information. On the other hand, the RAG system could be designed to manage the user queries and the retrieved content in a more modularized manner such that the language model could tell apart information from different sources.

Have you considered more sophisticated architectural approaches to information compartmentalization?

Yes, and we agree that these approaches represent a promising direction. For instance:

1. Multi-stream architectures: Employing separate attention streams for user queries and retrieved content might minimize unintended leakage.
2. Hierarchical processing: It involves structuring the flow of information within the model in distinct layers or stages, each dedicated to specific tasks, e.g. to interpret user queries or to retrieve/process/summarize datastore content. These preliminary ideas are part of our intended future research trajectory.

Final remarks

We appreciate your recognition of our empirical rigor and presentation clarity. While we acknowledge that our paper does not introduce fundamental theoretical contributions, we believe that our work makes a meaningful and timely impact by systematically studying an underexplored but critical vulnerability in RAG systems. Our findings serve as a reminder to the community of the trade-offs inherent in designing systems optimized for instruction-following while maintaining security.

Thank you for your detailed and thoughtful feedback. We value your insights and appreciate the opportunity to clarify our contributions. Below, we address each point raised in your review.

The core attack feels obvious. How do you respond to the criticism that this simply documents an obvious limitation?

While we acknowledge that the prompt-injection attack on RAG systems described seem intuitive, we respectfully argue that our work extends far beyond merely documenting a known issue or posting a blog in several key ways:

• Formal Problem Definition: We formally define prompt-injected data extraction, framing it as a generalizable threat rather than an isolated instance.
• Systematic Study: Our study is the first to systematically explore and quantify this vulnerability across a wide range of model scales and RAG configurations. This approach goes beyond anecdotal observations or blog posts by providing controlled experiments, comprehensive ablations, and formal problem definitions. For instance, we examine how variables like model size, context position, and chunking strategies directly influence extractability, offering new insights into these systems' susceptibility.
• New Insights: We especially highlight the interplay between position bias and vulnerability to prompt injection, identifying it as a critical factor influencing data leakage. This observation is novel and contributes meaningfully to the broader understanding of RAG vulnerabilities. Moreover, based on this finding, we propose the potential mitigation method which is proved to be effective.
• Practical Mitigations: We provide concrete, simple, and reproducible mitigation strategies and evaluate their efficacy, setting the stage for further work in more robust defense methods.
• Broader Implications: By testing production systems like GPTs, we demonstrate the practical significance of the vulnerability brought by strong instruction following abilities in real-world applications. This is not merely a hypothetical concern but a pressing issue for deployments where RAG systems are integrated into applications dealing with sensitive, confidential, or proprietary data., especially as larger models become more prevalent.

Similar to many other well-acknowledged empirical works in ML security [1,2,3,4,5], our study aims to emphasize the importance of a thorough exploration of a security issue, even though it is straightforward, and demonstrates baseline mitigations as practical starting points.

[1] Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., ... & Raffel, C. (2021). Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21) (pp. 2633-2650).

[2] Carlini, N., Hayes, J., Nasr, M., Jagielski, M., Sehwag, V., Tramer, F., ... & Wallace, E. (2023). Extracting training data from diffusion models. In 32nd USENIX Security Symposium (USENIX Security 23) (pp. 5253-5270).

[3] Nasr, M., Carlini, N., Hayase, J., Jagielski, M., Cooper, A. F., Ippolito, D., ... & Lee, K. (2023). Scalable extraction of training data from (production) language models. arXiv preprint arXiv:2311.17035.

[4] Huang, Y., Gupta, S., Zhong, Z., Li, K., & Chen, D. (2023). Privacy implications of retrieval-based language models. arXiv preprint arXiv:2305.14888.

[5] Lukas, N., Salem, A., Sim, R., Tople, S., Wutschitz, L., & Zanella-Béguelin, S. (2023, May). Analyzing leakage of personally identifiable information in language models. In 2023 IEEE Symposium on Security and Privacy (SP) (pp. 346-363). IEEE.

I have considered your comments and rebuttal, and I still believe that this paper is a relatively marginal contribution, which does deserve an acceptance but not necessarily a spotlight or oral.

As such, I keep my main score (slightly raised the others), but I further urge the PC/SAC/AC to accept this paper if they feel "on the fence" about it. I'd give a 6.5 or 7 if I could.

Thank you very much for your comments and advice. We appreciate the opportunity to clarify and strengthen our paper based on this feedback. Here we respond to each point:

The paper does not seem to introduce any novel mitigation strategies.

We acknowledge your concerns about the two mitigation strategies. However, 1) we are one of the first to point out the relationship between context use and vulnerability to prompt injection attack, based on which we introduce and show the effectiveness of the corresponding mitigation method built upon PINE, and 2) the primary contribution of our paper is not in proposing new methods but in systematically identifying and analyzing a significant security vulnerability in RAG systems. Similar to other works in ML security [1,2,3], our study emphasizes the importance of a thorough exploration of a security issue and demonstrates baseline mitigations as a practical starting point. We consider a thorough analysis of a safety problem to be a complete study and provide foundational insights for future study, and advanced mitigation techniques could be an independent extension work.

[1] Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., ... & Raffel, C. (2021). Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21) (pp. 2633-2650).

[2] Carlini, N., Hayes, J., Nasr, M., Jagielski, M., Sehwag, V., Tramer, F., ... & Wallace, E. (2023). Extracting training data from diffusion models. In 32nd USENIX Security Symposium (USENIX Security 23) (pp. 5253-5270).

[3] Nasr, M., Carlini, N., Hayase, J., Jagielski, M., Cooper, A. F., Ippolito, D., ... & Lee, K. (2023). Scalable extraction of training data from (production) language models. arXiv preprint arXiv:2311.17035.

The paper does not investigate whether the choice of indexing and retrieval mechanism would affect the attack results.

Thank you for emphasizing the potential impact of indexing and retrieval mechanisms on attack outcomes. We fully agree that expanding our analysis to include alternative RAG architectures would enhance the scope and depth of our findings. This is an avenue we plan to explore in future work.

In this submission, however, we intentionally focused on the simple but representative and reproducible RAG setups to prioritize scalability and reproducibility and to investigate how vulnerabilities evolve with model size. As noted in the introduction, we centered our study on a widely-adopted and established RAG method, the retrieve-in-context (RIC) manner, which is adopted as a canonical setting in various real-world RAG systems like LlamaIndex and LangChain. While many diverse architectures exist, they ultimately rely on LLMs as their core component and, as such, remain fundamentally vulnerable to prompt-injection attacks—a point we aim to illustrate in this work.

Readability of some paragraphs (e.g., line 47 to 67 and line 452 to 476) is low due to their lengthy nature. I suggest breaking them into separate paragraphs or shortening some sentences.

We appreciate your feedback on readability. The sections you mentioned (lines 47–67 and 452–476) will be revised to improve clarity and conciseness. We will divide lengthy passages into shorter paragraphs and simplify complex sentences where appropriate.

For the results and findings presented in the paper, were you performing attacks on naive RAG?

Yes, as stated in the introduction, our experiments were conducted on RAG systems using the widely-adopted RIC method, i.e. naive RAG. We deliberately chose this setup to highlight vulnerabilities in a straightforward and representative implementation. This choice helps ensure the reproducibility and applicability of our results across a broad range of RAG systems employing similar designs.

What will be the attack results when you apply the prompt-injected data extraction attacks on GraphRAG?

We acknowledge the value of exploring attacks on GraphRAG and other SOTA RAG architectures. While our study emphasizes the vulnerability of naive RAG systems, the fundamental principles of prompt-injection attacks and their exacerbation with model scaling should extend to other configurations as long as there are LLMs engaged, including GraphRAG, but the attack might be less effective since GraphRAG includes a summarization stage that preprocesses the text sources and thus some information could not be copied verbatim. We plan to conduct additional experiments in this direction and hypothesize that systems integrating advanced graph-based indexing might still inherit similar vulnerabilities due to their reliance on LLMs.

We appreciate your thoughtful feedback and provide responses to the identified weaknesses and questions below:

The experiment setup might be overly simplistic

We acknowledge the concern regarding the simplicity of the setup. In our study, we aimed to demonstrate the core vulnerability in a controlled manner to isolate and analyze the factors contributing to data extraction risks, thus deliberately choosing the simplest RAG design. We agree that exploring more sophisticated RAG setups, such as those involving structured prompts that separate system and user inputs more explicitly, would strengthen the paper and introduce more insights. Future iterations of this work will incorporate advanced prompting techniques and more robust setups as new mitigation strategies.

Scenarios where data extraction is a concern

We appreciate the suggestion to expand on real-world scenarios. The risk of data extraction becomes particularly concerning in scenarios where RAG systems are integrated into applications dealing with sensitive, confidential, or proprietary data. For instance:

• Enterprise Data Systems: RAG systems are often used in corporate environments where datastores include trade secrets, internal documentation, or confidential client information. An adversary extracting such data could lead to significant financial losses, breaches of confidentiality agreements, and reputational damage.
• Healthcare and Finance Applications: In sectors like healthcare and finance, RAG systems may retrieve patient records, financial statements, or other regulated information. The unauthorized disclosure of such data could violate privacy laws (e.g., GDPR in Europe) and expose institutions to legal and financial penalties.
• Consumer Applications: Personalized applications, such as virtual assistants or customer service bots, may access user-uploaded files or personal data. Extraction attacks here could expose users to identity theft or other forms of exploitation, especially if sensitive information (e.g., passwords, addresses) is stored in the datastore.

Did you confirm with model provider that the Wikipedia articles are not included in the model training? Also, newer articles do not necessarily mean they are not overlapping with content in existing Wikipedia, so it’s a good idea to run a quick n-gram overlap check to remove articles that are not entirely new.

We selected Wikipedia articles created after November 1, 2023, to minimize overlap with the models’ training data. This cutoff date was chosen because most of the models tested in our study are unlikely to have been trained on data updated beyond this timeframe. This approach has also been validated by prior published work, such as Shi et al. [1], which similarly used recent and obsolete wikipedia data to construct benchmarks for membership inference attacks. Admittedly, this ensures minimal overlap but does not guarantee absolute exclusion of overlapping content, but we consider this to be a reasonable assumption and have minimal effects on obtaining meaningful experiment results.

[1] Shi, W., Ajith, A., Xia, M., Huang, Y., Liu, D., Blevins, T., ... & Zettlemoyer, L. (2023). Detecting pretraining data from large language models. arXiv preprint arXiv:2310.16789.

Did you evaluate whether PINE affects RAG QA performance?

No, because PINE work has already done such experiments in the “ Results on Retrieval-Augmented Question-Answering” section, and the results show that PINE is performing better than baselines.

what if we don’t have access to the GPTs’ system prompt (also leaked)?

In our work, the GPTs are attacked in a two-stage manner, where in the first stage, we obtain the system prompt by simple prompt injection attack, and then in the second stage, we leverage the information in the system prompt to attack it. In future scenarios where the system prompt of GPTs is not directly accessible, the attack may require adjustments tailoring to the specific design of GPTs or other information. As our attack on GPTs is just a proof-of-concept to show how instruction following abilities could be leveraged to leak data, we did not dig further into such cases.