Adversarial Search Engine Optimization for Large Language Models

We are glad that the reviewer believes that the threat model in our work is novel and that it offers new insights in the study of LLM security.

More defenses on the LLM itself (e.g., maybe safe alignment) may have a good result on defending the Preference Manipulation Attacks.

We agree that defenses can help mitigate against preference manipulation attacks. However, it is not possible for us to test them, as the systems we attack are full black-box.
Furthermore, while we expect defenses to mitigate against our attacks, we believe that it’s unlikely that they are going to solve the problem: as shown by $1$ , most prompt injection defenses (including fine-tuning related defenses such as the instruction hierarchy $2$ ) have limited effectiveness against indirect prompt injection attacks. Thus, an attacker could first spend a lot of work finding a successful attack (e.g., by using dummy web pages), and then deploy this attack at scale on targeted web pages.

more application scenarios with adopting these attack can set a border future scope for this proposed attacks

We agree with the reviewer that these attacks have the potential to be adopted for other uses, however we believe that this is outside the scope of our work.

References:

$1$ https://arxiv.org/abs/2406.13352

$2$ https://arxiv.org/abs/2404.13208

We are glad that the reviewer found our paper forward-looking, well written, and our experimental validation considerable.

It is plausible that a well-designed LLM-powered search engine could avoid these problems.

While we believe that a well-designed LLM-powered search-engine could mitigate these problems, no prompt injection defense proposed so far has shown to be bullet-proof and to be a potential solution to avoid these problems altogether. We agree with the reviewer that we hope that our work will spark more research in this area, and we believe that this is one of the main reasons for the relevance of our work.

We thank the reviewer for the feedback, and we are glad that they find our paper timely, the experimental design careful, and that they endorse our approach to evaluating the attacks, and that they appreciate the real-world implications of our work. We address their concerns as follows.

Given that this paper does not introduce any novel attack algorithms, it is better regarded as a measurement paper $...$ the number of attack scenarios employed is limited

We agree with the reviewer that our work is indeed a measurement paper: we demonstrate and quantify how preference optimization attacks affect real production systems. Moreover, we introduce and study the game-theoretic dynamics that stem from multiple adversaries launching competing prompt injection attacks.
Regarding the limited number of attack scenarios, it would of course always be better to have more but we are faced with an inherent engineering limit due to the time it takes for pages to be indexed by the search engine. Furthermore, we were worried that adding too many adversarial pages could cause our lab’s website to be heavily penalized (or even delisted) on search engines.

Nonetheless, we show the effectiveness of our attacks in diverse settings: we evaluate different LLM uses (i.e., product reviews, news aggregation, and plug-ins), scenarios where multiple attackers are in place (Section 5.2), with attacks on third party websites (Section 5.3), with different prompts (Section 5.4), with fake products competing against well-reviewed products, and with more “natural” prompts on pages that are at the top of the search index (Section 5.1).

If the reviewer has additional suggestions on how to scale the number of experiments further, we would be happy to try to provide more results.

I am curious whether open-source LLMs, combined with RAG, could be used to simulate attack scenarios, thereby facilitating further discussion, such as the exploration of various defense mechanisms.

Simulating a defense with an open weights LLM combined with RAG is a potential way to simulate these systems to then check the effectiveness of defenses. However, 1) we do not have access to an open source search index that we can use to feed the RAG and the LLM, and 2) we do not know how production systems employ RAG, or which type of RAG they use.
We believe that these two issues would make testing defenses not particularly insightful or relevant as the results in the real-world systems could be significantly different.

Furthermore, we know that we can expect these defenses to somewhat work against our attacks. However, we also know that none of them is going to bring the ASR to 0% (as none of the defenses claims to do so, and attacks still succeed to a certain extent). Thus, having a defense in place would simply shift the problem to first finding a prompt injection that works against the given defense (e.g., using dummy websites), and then deploying it at scale on the targeted website.
We believe this problem is orthogonal to our paper.

so comparing different attack algorithms would be highly valuable

There are three reasons why we do not compare different attack algorithms:

1. As mentioned above, adding more web pages is not practical due to the time it takes to create the pages, indexing, and potential consequences on our lab’s web domain.
2. We believe that most of the value of our work is in establishing the threat model, showing that these attacks work in the first place, and evaluating the dynamics involving multiple attackers.
3. Finally, finding prompt injection attacks against black-box systems remains a rather ad-hoc process today. And so there isn’t really any “collection” of attacks we could try. We experimented with a few standard prompt techniques employed by past, successful prompt injections, and found that these were sufficient for our purposes.

We thank the reviewer for the feedback and we are glad that they find our attack novel, the narrative and the presentation clear, and that our work provides several significant insights. We also thank the reviewer for their suggestions regarding the abstract and for spotting the typo in the caption of Fig. 5 (b). We fixed both in the updated PDF.

Table 1 is not entirely clear – how is the recommendation rate calculated? Should the percentages sum to 100%?

The frequency of being recommended is the number of times a product is recommended, divided by the number of times we present it as a candidate to the LLM. The reason why it does not sum to 100% is that the LLM could recommend one product, multiple products, or no product at all. We clarified this in our revised PDF in Section 4, paragraph “Metrics”.

Am I understanding Fig 5 to mean that Claude 3 Opus never chooses the plugin with “no attack”? Is there any particular underlying reason that GPT-4 and Claude have such different behaviour?

Your understanding is correct. Our intuition is that it could be a consequence of different preference and safety training techniques/datasets/parameters, which could make Claude prefer the source which claims to give balanced results from multiple providers over the one without attack, which claims to include news from one news provider only.

it seems to me at least that this sort of prompt engineering is very fickle to the model type/version

We agree that prompt injection attacks are, at this time, a bit of a trial and error mechanism. However, once a successful attack strategy is found it can easily be deployed at scale across many websites. The aim of our paper is to show that these attacks work on production systems despite the low effort we put into finding effective attacks, with little prompt engineering. We would expect a more systematic attack to have even better and transferable results.

We thank the reviewer for the feedback. We are glad that they find our experiments convincing evidence that SOTA LLM-based recommendation systems are vulnerable to prompt injections. We now address the concerns of the reviewer:

If a product X is not well-known $...$ SEO seems much more important as X has to be one of the options in the LLM search database.

Indeed, our attack is not a substitute for traditional SEO, but rather complementary to it. Our attack is not meant to get the product ranking 100th to be recommended to a user, but the product ranked 10th or even 2nd to the first position. Let’s consider for example the GitHub experiment in our paper in Section 5.1, where we consider our lab’s GitHub page, HuggingFace page, and website as competitors: the prompt injection in our lab’s GitHub page makes the LLM recommend only to visit our GitHub page, which would normally rank 2nd or 3rd.

if X is well-known $...$ the established company would suffer from losing its reputation by conducting this attack

This is indeed a valid concern. As a matter of fact, after the disclosure of our results, Microsoft added to their “Webmaster guidelines” $1$ that prompt injections “can lead to demotion or even delisting of your website from our search results.” However, attacks can sometimes be stealthy and hard to detect. For example, in our paper we show that attacks can be in third-party websites, or not look like attacks at all. In both cases, a reputable website would have plausible deniability.
This may lead to a situation similar to traditional (adversarial) SEO: there are practices that are prohibited, but some companies still try to abuse them without being punished.
In this sense, we believe that our work is important as it raises awareness about the potential of these attacks, and fosters the deployment of defenses and/or changes to policies as noted above.
Finally, we note that final users might not even necessarily realize that there are attacks in place: especially for attacks that do not look like prompt injections, we believe that the reputational damage would be extremely limited. And this would be even more the case if such attacks become common practice among websites.

some experiments with prompting defense would be helpful to show if the threat could be easily mitigated.

We believe that it should not be the user’s responsibility to defend against these attacks, but rather the LLM provider’s. But it is conceivable that the LLM provider could add additional system prompts to mitigate prompt injections. Unfortunately we cannot test this with proprietary models where we can only modify the user prompt. In any case, prompting-only defenses (even when highlighting the untrusted data with special delimiters) are known to be brittle $2$ .

References

$1$ https://www.bing.com/webmasters/help/webmaster-guidelines-30fba23a

$2$ https://arxiv.org/abs/2406.13352

Right, we can do this with some proprietary models. But to our knowledge there is no way to do this with proprietary search engines. Eg Bing or perplexity (even if they might use OpenAI models under the hood) don't let you specify a system prompt. Moreover, since these systems pass the retrieved data directly to the LLM, we also cannot implement the Chen et al. défense.

This could work if we built our own LLM search engine around a proprietary LLM, but it is unclear how to design such a system to be faithful to proprietary designs with a proprietary search index.

We are glad that the reviewer found the problem we address in our paper “novel and highly relevant”, the threat model we consider “strong”, and the experiments “very well designed”.

We note that poisoning the Web as in the Carlini et al. paper is rather orthogonal to our work, as they consider poisoning model “pre-training” rather than a test-time RAG setup as we do.

But we certainly agree that prompt injections (and LLM manipulation attacks more broadly) will affect any system where LLMs have to process untrusted text.
Our aim here is not to show new attack avenues, but rather to demonstrate how these attacks (despite being known for over a year) can affect production systems with significant consequences for the billion-dollar SEO industry.
Our technical contributions are: (1) a systematic and quantitative evaluation of these attacks, which were so far only known as proof-of-concepts (e.g., in the work of Greshake et al., with Bing in side mode); and (2) the discovery and evaluation of the game theoretic aspects of “multi-party prompt injections”.

The latter is extremely relevant in the search engine setting, where there are multiple parties competing for the best spot in the results returned by the chatbot or the search engine. To the best of our knowledge, ours is the first work covering the game-theoretic aspects of competing prompt injection attacks.

Overall, we believe our work is valuable to show that LLM manipulation attacks are a risk that production search engine companies should take seriously. It is thus encouraging that Microsoft has made changes to Bing in response to our work, to mitigate these attacks.