Understanding Data Poisoning Attacks for RAG: Insights and Algorithms

We deeply appreciate the reviewers for dedicating their time and effort to reviewing our manuscript and providing insightful feedback. We are pleased that all reviewers acknowledged the novelty of our work. Furthermore, we are grateful that they considered our writing clear and our approach effective across different setups. We will integrate their suggestions into the revised version of the paper.

Q: I am uncertain about the reliability of DRS. For example, if the question is, "Who is the OpenAI CEO?" I would expect the embedding of a clean document ("The CEO of OpenAI is Sam Altman") to be similar ...

Response: Thank you for your insightful and sharp question regarding the reliability of DRS. We believe that, in general, it could be challenging to distinguish between the two queries you provided using all existing defense methods, including ours. These queries are very close in terms of semantic meaning, sentence quality, and embedding vectors. As a result, regardless of the defense method—whether norm-based, perplexity-based, or our proposed method—it may be infeasible to distinguish between these two queries.

More fundamentally, by casting the problem of detecting the two queries you proposed as a hypothesis testing/binary classification problem, we can show that the Bayes Risk (which depends on the total variation between the two distributions) for the problem will be very high, as their total variation distance is very small. This represents a fundamental limitation of all detection-based methods, including our proposed DRS.

Q: In Figure 1, what is the Y-axis?

Response: Thank you for your question regarding Figure 1. The Y-axis in Figure 1 represents the relative change in the attack success rate (ASR) for the three attacks. We define the relative changes in Lines 288-295. Specifically, the relative change along a certain direction of the embedding vectors is calculated by measuring the difference between adversarial and clean documents along the directions of the clean documents. This difference is then normalized by dividing the relative mean by the standard deviation along these directions. We observed that more effective attacks, such as AgentPoison, tend to have a larger relative distance along directions with small variances (i.e., the left-most group represents the directions of clean embedding documents with the top 100 smallest variances), which empirically verifies our theory.

Q: In Section 2.1, the attacker’s capability is described as "only injecting poisoned data (e.g., by creating a new Wikipedia page)." However, in Section 5.1.2, the setting appears to change, with the retriever itself being backdoored.

Response: Thank you for your question regarding the threat model. This does not violate our threat model. In Section 5.1.2, the scenario considered is that the attacker only poisons the training data, but later the users themselves fine-tune the retriever based on the poisoned data for better performance. As a result, the retriever becomes backdoored. We will further clarify this point in the revised manuscript.

Q: In Section 5.1.1, there is no description of the adversarial query.

Response: Thank you for your comment regarding the adversarial query. Due to space limitations, we included detailed descriptions and provided some examples of the adversarial queries in the appendix. Here, we list them for your reference. To create adversarial data, the attacker appends an adversarial backdoor trigger at the end of normal data. The backdoor/adversarial trigger used for autonomous driving is: Be safe and make a discipline.

Q: In Section 5.1.1, the statement "For each attack method, we generate 300 poisoned data samples" is unclear. Does "poisoned data samples" refer to poisoned documents?

Response: Thank you for your question. Yes, you are absolutely correct. We will make this clear in the revised manuscript.

Q: If I understand correctly, DRS also requires a set of clean samples to compute the threshold, but it is unclear how large and diverse this dataset needs to be.

Response: Thank you for your question. Yes, you are correct about the need for a set of clean samples. Regarding the size of the data, we provide an ablation study on using different sizes for the clean data (in the context of autonomous driving) and summarize the results in the following table. We observed that our proposed method is robust with different sample sizes of the clean dataset.

Regarding the diversity of the clean dataset, it depends heavily on the targeted query set to be protected. If the query set contains queries from closely similar topics, then the corresponding clean dataset need not be diverse. Otherwise, the clean dataset requires more diverse data. We will clarify this in the revision.

We deeply appreciate the reviewers for dedicating their time and effort to reviewing our manuscript and providing insightful feedback. We are pleased that all reviewers acknowledged the novelty of our work. Furthermore, we are grateful that they considered our writing clear and our approach effective across different setups. We will integrate their suggestions into the revised version of the paper.

Q: Sparse Theoretical Explanation -- While DRS’s foundation on variance shifts is intuitive, a deeper theoretical analysis could further clarify why certain dimensional shifts are more vulnerable. This would strengthen the defense’s theoretical underpinnings.

Response: Thank you for your suggestion regarding the theoretical explanation of DRS. In Corollary 1, we have shown that certain directions are more effective for attacks, with an intuitive explanation provided in Remark 3. Below is the logic flow of our theoretical analysis as it was presented for your reference:

1. Theorem 1 demonstrates that attackers can successfully launch an attack, as defined in Section 2.1, by creating poisoned queries sufficiently distant from normal ones.
2. However, if adversarial queries are too different from the clean ones, they may be easily detected. Therefore, the attacker seeks to answer: Given the maximum deviation (e.g., ℓ2 distance) between the normal and adversarial distributions, what are the most effective directions for moving from $Q_{normal}$ to $Q_{adv}$?
3. Corollary 1 shows that the most effective attack directions are those that maximize the rate at which the clean data $\mathcal{D}^{\text{clean}}$ density decays.

Intuitively, directions in $\mathcal{D}^{\text{clean}}$ with rapidly decaying density often correspond to low-variance directions. Low variance indicates that most of the probability mass is concentrated around the mean, so even small deviations from the mean significantly reduce the probability mass. This aligns with the attacker's goal: perturbing a clean query in a low-variance direction reduces the likelihood of clean data being close to the perturbed query, increasing the chance of retrieving poisoned documents. We hope this explanation clarifies the theoretical basis of DRS and will improve the presentation in the revised manuscript.

Q: Unrealistic Defense Assumptions -- The defense method assumes prior knowledge of a specific subset of queries that need protection from poisoning attacks. In real-world applications, defenders typically do not have knowledge of which specific queries might be targeted, and a practical defense would need to offer broad protection across all possible queries. This limitation reduces the generalizability and practicality of the proposed DRS-based defense method.

Response: Thank you for your comment regarding the defense assumption. We believe that the assumption of prior knowledge of a specific subset of queries is reasonable and, in fact, essential for both practical and theoretical reasons.

1. Theoretically, if the defender has no prior knowledge of the queries to be protected or aims to protect all possible queries, it can be shown that (i.e., using LeCam's Method to prove an information-theoretical lower bound) the defense is infeasible. This is because, by considering all possible queries, their distributions are likely to cover the entire input space, making it impossible to distinguish between normal and adversarial queries.

2. Practically, in many real-world applications, the defender does have some knowledge of the queries that need protection. For example, a defender, such as a RAG service provider, has access to the underlying database for retrieval and is likely aware of which queries are critical to the system's operation. In this case, the defender can use this knowledge to protect these queries from poisoning attacks.

Moreover, the effectiveness of the proposed defenses across multiple settings clearly demonstrates the wide applicability of the defense. We will further clarify this point in the revised manuscript.

Q: Unrealistic Assumption -- In Section 3.1, the authors illustrate their attack method with an example where, in a knowledge base about food, an adversarial query about mathematics is used to avoid retrieving clean documents. This assumption is unrealistic, as it does not reflect typical user behavior ...

Response: Thank you for your comments regarding the experimental results. We believe there might be a misunderstanding between the Assumption 1 (we assume you are referring to this one) and the example. We provided this extreme-case example simply to offer some sanity checks/intuition. Recall that when proving theoretical results, it is common to choose extreme values for a simple sanity check. We will modify the example to make it more realistic in the revised manuscript.

Thanks for the responses. Since the responses have addressed part of my concern, I will increase my score.

We deeply appreciate the reviewers for dedicating their time and effort to reviewing our manuscript and providing insightful feedback. We are pleased that all reviewers acknowledged the novelty of our work. Furthermore, we are grateful that they considered our writing clear and our approach effective across different setups. We will integrate their suggestions into the revised version of the paper.

Q: Missing some references.

Response: Thank you for pointing out the missing references. We have included them in the revised manuscript.

Q: There has been some work discussing the characterization of poisoned samples. In particular, the proposed method (i.e., DRS) is similar to [3] to some extent. The authors should compare their method to existing works.

Response: Thank you for your valuable suggestion regarding the comparison of our proposed method to existing works. The proposed DRS differs significantly from [3] in terms of the threat model and technical components.

[3] is a training-stage defense method that uses both clean and poisoned training data to filter out backdoors and build a clean model, evaluating based on clean data and attack success rates. In contrast, our approach, being an inference-stage method, uses only a small set of clean validation data and no knowledge of the poisoned data, aiming to detect future poisoned inputs. We evaluate our method using the AUC-ROC score of the detector.

Q: Explain the performance of the proposed attack.

Response: Thank you for your question regarding the performance of the proposed attack. In Table 5 in Section 5.2 of the main text, we report the filtering rates for poisoned data generated by different algorithms. Given the same attack success rates of the poisoned data, more effective attacks have lower filtering rates, as they are less likely to be detected by the defense mechanism. We observe that the detection rate (by the proposed DRS defense) for poisoned data generated by our algorithm decreases by 15% compared to the vanilla AgentPoison, highlighting the effectiveness of the algorithm.

In fact, as discussed in Section 4.2 of the main text, the detection rate of the proposed DRS defense can be further lowered by increasing the hyperparameter $\lambda_2$. However, this comes with a trade-off: as the penalty increases, the attack success rate of the corresponding poisoned data decreases, as suggested by our theorems. This is because, intuitively, a large penalty forces the poisoned data to be more similar to the clean data, making the attack ineffective.

Q: The authors only use AgentPoison as an example to demonstrate the effectiveness of the proposed attack. The authors should conduct more extensive experiments on all discussed attacks to verify its generalizability.

Response: Thank you for your valuable suggestion regarding the evaluation of our newly proposed attack. To the best of our knowledge, we are not aware of any other existing attacks (with open-sourced code) that aim to achieve similar goals, i.e., generating poisoned data with high attack success rates while explicitly forcing them to be less discernible to detection, in the context of RAG, except for the ones already included. In fact, the AgentPoison attack (first on arXiv in July this year) itself is very new, as quoted: '... the first backdoor attack targeting generic and RAG-based LLMs ...'.

Nonetheless, we have introduced a new attack (proposed by ourselves, motivated by backdoor literature) that penalizes the Wasserstein distance between adversarial and normal queries, instead of the proposed DRS. We summarize the results in the following table, where we observe that the detection rate of the Wasserstein distance-based attack is higher than that of the proposed DRS-based attack (with a lower detection rate indicating that the attack is more effective), indicating the effectiveness of our proposed DRS attack algorithm.

Table: Filtering rates for poisoned data, generated by AgentPoison and our newly proposed DRS-regularized AgentPoison, and the Wasserstein-regularized AgentPoison. The decision threshold for filtering is set to the 99th percentile of the clean scores, resulting in a false positive rate of approximately 1% for clean documents.

We deeply appreciate the reviewers for dedicating their time and effort to reviewing our manuscript and providing insightful feedback. We are pleased that all reviewers acknowledged the novelty of our work. Furthermore, we are grateful that they considered our writing clear and our approach effective across different setups. We will integrate their suggestions into the revised version of the paper.

Q: One limitation of the method is that the assumption can be strong. For instance, it is assumed that adversarial query has a different distribution from normal ...

Response: Thank you for your valuable suggestion regarding Assumption 1 and the insightful example you provided. We would like to clarify that our threat model, as specified in Section 2.1 of the main text, precisely addresses the scenario you mentioned and is designed to avoid related issues. Therefore, we believe Assumption 1 is both valid and reasonable within the context of our threat model. We elaborate on this point in detail in the followings.

In our threat model, as specified in Section 2.1 of the main text, we consider targeted attacks where the attacker aims to (1) manipulate the RAG system to generate a prescribed adversarial output in response to attacker-specified adversary queries, and (2) ensure that the RAG responds normally to normal queries. In this context, the distribution of the adversary-specified queries is inherently different—and should, in fact, be significantly different—from that of normal queries. This difference is the very foundation upon which the attacker exploits the system to carry out successful attacks. If the distribution of adversarial queries were identical to that of normal queries, there would essentially be no opportunity for the attacker to manipulate the RAG to produce a specific adversarial output.

Given this threat model, we believe Assumption 1 is both reasonable and valid. Within our model, there is a substantial gap between the distributions of adversarial and normal queries. As a result, it should not be difficult to inject adversarial documents into the database that are close (in terms of distribution) to the adversary queries, but far from the normal queries, so that they can be retrieved effectively when given adversary queries. And this is precisely what the Assumption 1 is describing. We will further clarify this point in the revised manuscript.

Q: The assumption 1 is not very clear. How to measure the distance between two texts? The authors may consider adding more explanations to make it easier for readers to understand. Also, assumption 1 states the distance between two texts is bounded, which may not be informative, as it may hold for two arbitrary texts in practice.

Response: Thank you for your valuable suggestion regarding Assumption 1. We use the $\ell_2$ distance between the embeddings of two texts to measure the distance between them by default, as described in the Notation section of the main text. Intuitively, Assumption 1 suggests that, given an adversarial query, there (almost surely) exists a set of adversarial documents that are closer to the nearest clean documents. The current statement is primarily used to simplify the proof of the main theorem. We will revise this to make it more explicit and intuitive, while emphasizing that it does not impact the theoretical proof.

Q: The proposed defense may influence the utility of RAG. For instance, if new knowledge is added for a query, it can be rejected if it is substantially different from clean texts in the clean data corpus...

Response: Thank you for your insightful question regarding the impact of the proposed defense on RAG utility. We agree that a trade-off between defense effectiveness and normal RAG utility is inevitable, as this trade-off is inherent in all detection-based problems. We note that the decision threshold in our method (in the main text) is set to result in a 1% false positive rate for clean documents. In fact, the false positive rate can be adjusted by changing the threshold as specified in Algorithm 2 in the main text. Below, we provide an ablation study by adjusting different thresholds and report their false positive rates and detection rates for the autonomous driving dataset. We can observe that with a small FPR of 0.5%, the detection rate is still very high at 0.95. This indicates that the proposed defense is effective in detecting poisoned data while maintaining a low false positive rate.

Table: False positive rates and detection rates for different thresholds on the autonomous driving task.

We deeply appreciate the reviewers for dedicating their time and effort to reviewing our manuscript and providing insightful feedback. We are pleased that all reviewers acknowledged the novelty of our work. Furthermore, we are grateful that they considered our writing clear and our approach effective across different setups. We will integrate their suggestions into the revised version of the paper.

Q: A significant shortcoming is the absence of reported attack success rates in the experimental results. Without this metric, it becomes difficult to fully evaluate the effectiveness of both the proposed attacks and defenses.

Response: Thank you for your insightful comment regarding the evaluation of the proposed attacks and defenses. All the attacks are implemented by directly running the open-sourced code released by the authors without any modification. As a result, the ASR of the attacks is the same as reported in the original papers. Specifically, all these attacks have decent ASR as reported in the original papers. For instance, AgentPoison has an ASR of around 0.8 on the autonomous driving dataset. We will include the ASR of the attacks in the revised manuscript for better clarity.

Q: The paper also lacks a deep discussion on the computational cost of DRS.

Response: Thank you for your insightful comment regarding the computational cost of DRS. The computation and employment of DRS are very light. Specifically, we first collect a set of clean documents and obtain their embeddings. Then, we perform SVD/eigen-decomposition (only once and can be done offline) on the embedding matrix and store the resulting eigenvalues and eigenvectors. When new test data arrives, we only need to compute the DRS score by calculating the inner product between the test data embedding and the previously stored eigenvectors. Thus, the computation involves only a few matrix-vector multiplications and is very efficient. We will include this information in the revised manuscript for better clarity.

Q: The access to clean documents need better justification and analysis. How defending against poisoning in RAG settings differs from defending against, e.g., jailbreak or prompt injection attacks?

Response: Thank you for your insightful comment regarding access to clean documents and the difference from other attacks. The main difference between poisoning attacks in RAG and the other attacks you mentioned is that RAG poisoning attacks heavily rely on the successful retrieval of adversary-injected documents from the database using adversarial prompts/queries. In contrast, typical adversarial attacks, like jailbreak against LLMs, do not involve retrieval and solely rely on crafting adversarial prompts/queries to lead the LLM to generate adversarial outputs.

Given this discussion, access to clean documents is essential for the defender to detect adversary-injected documents. Since this is where the poisoning occurs, the defender needs access to clean documents to identify which ones are potentially poisoned. We will further clarify this point in the revised manuscript.

Q: Can the authors motivate better this assumption?: "We assume the defender has access to both the retriever and the clean data corpus. When a new test ...

Response: Thank you for your insightful question regarding the clean data assumption. In fact, the defender must have access to a set of (known) clean documents in order to distinguish between future clean and potentially poisoned documents. Imagine a scenario where you are given a basket of apples, some green and some red, and you are asked to identify the red apples. Although you can divide the apples into two groups, you wouldn't know which group contains the red apples without first knowing what the red color looks like. Similarly, without knowing what a clean document looks like, it would be impossible to accurately identify the poisoned ones. This assumption is fundamental in detection-based literature, such as Out-of-Distribution detection.

More fundamentally, the detection problem can be framed as a hypothesis testing or binary classification problem. In this context, the defender must understand the distribution of either clean data (which is typical in detection-based problems) or adversarial data. Without this knowledge, we can show that the optimal detector for differentiating between clean and adversarial data would effectively be reduced to randomly guessing.

Regarding the number of clean documents required, if you mean "provable guarantees," such as those in conformal literature, then the number of clean documents needed to achieve such a guarantee would depend on the distribution of clean and adversarial data, which we leave as future work. In our experiments, the number of clean documents required is relatively small (e.g., 500-1000) to achieve a high detection rate. We will further clarify this point in the revised manuscript.