Tensor Trust: Interpretable Prompt Injection Attacks from an Online Game

Thank you for your review. We’re glad to hear you found our game to be well-designed and useful as an adversarial robustness benchmark! We've uploaded a new revision of the paper with changes summarized in the top-level shared response. We'd also like to offer a clarification on prompt extraction:

The task of prompt extraction is distinct from the Trust Tensor. In other words, the Trust Tensor is not very suitable for collecting the prompt extraction dataset.

You're right that the game does not explicitly reward players for performing prompt extraction, but many players found that an effective way to achieve "access granted" was to first perform prompt extraction, read the password from the prompt, and then enter the password verbatim. See the middle row of Figure 1 for an example of this. Around 27% of successful attacks on other players contain the defender's exact access code (indicating that the attacker performed successful prompt extraction), and the prompt extraction robustness benchmark contains a subset of ​​569 strong, human-verified examples of prompt extraction across different defenses. (The number 569 has increased from the original submission, as we expanded and manually reviewed our benchmark subsets for the latest revision.)

Thank you for your review and suggestions for improvement. We’ll address these points one by one below. We've also uploaded a new revision of the paper with various general improvements (see shared top-level response). We hope we have addressed all your concerns; please consider revising your score if this is the case.

Weakness 1: Game mechanism

[1] Although the size of the dataset is pretty large, the mechanism of this game is monotonous.

We're not sure whether "monotonous" in this context means "not enjoyable" or "generates uninteresting data", so we'll discuss both:

• Do players like it? Tensor Trust is a simple game, but players appear to enjoy it. After our initial promotion push, much of our web traffic has been driven by organic sharing on social media (i.e. not due to our own promotion), which continues to drive a few thousand attacks per day. As of November 16th, we have >3,000 registered accounts and >330,000 submitted attacks, as well as >250 players on our Discord, which shows strong engagement.
• Is the data interesting? We chose a simple objective for the game (make the LLM say "access granted") so that it would be easy to objectively evaluate attack success (as opposed to, e.g., jailbreaking attacks, which require subjective evaluation). Despite its simplicity, we've found that this objective is sufficient to elicit a rich dataset of attacker behaviors, including prompt extraction as a means to gain additional information (as opposed to just prompt hijacking), and the various classes of attack outlined in Table 1. Overall, we found that the game setup strikes a good balance between making it easy to do objective evaluation of attack/defense strategies and encouraging diverse techniques.

Weakness 2: Relevance to ICLR

[2] I am unsure whether the topic of this paper aligns with the theme of this conference.

We think that ICLR would be a good venue for this paper due to its history of publishing strong benchmark datasets and adversarial robustness work. In the past, ICLR has been a venue for seminal work in this area, such as the ImageNet-C dataset for robust computer vision (ICLR'19, now cited 2.6k times), or this popular vision-based adversarial example defense (ICLR'18, now cited >10k times). More recently, ICLR 2023 published a variety of papers that consider datasets/benchmarks (examples: [1], [2], [3]) or adversarial robustness (examples: [1], [2], [3]). It also had two workshops on robustness/security/privacy ([1], [2]). Both "societal considerations including fairness, safety, privacy" and "datasets and benchmarks" are listed as relevant topics in this year's CFP, so we expect this kind of work will continue to be of interest to the broader community at ICLR 2024.

Question 1: Applications

[1] What are the potential applications of this dataset? In other words, how can it help improve the robustness of LLMs?

This is a good point, and we've modified the paper to make this more clear. To summarize, here are some ways in which the data and experiments in the paper can help improve LLM robustness:

• Researchers can use our robustness benchmarks (section 3/section 6) to measure whether newly released LLMs (or new forms of LLM finetuning) improve robustness to prompt hijacking and prompt extraction.
• Our taxonomy of strategies (section 4) and qualitative analysis in the appendix shows various ways in which existing LLMs fail to be robust to prompt injection, which we believe is useful to target future efforts in fine-tuning LLMs against prompt injection (e.g. the analysis in appendix H shows that system-user message roles have little effect on GPT 3.5 Turbo, which highlights the need for stronger steerability for these types of models). This qualitative analysis may also be useful for prompt engineers and red-teaming efforts.
• We also hope that our data will be useful for directly training better-defended models (by optimizing them to ignore the adversarial examples in our dataset), as well as for automated red-teaming (e.g. learning multi-step attack strategies from the data). We leave these more complex applications to future work.

We've now improved the contribution statement in the introduction to make the first two contributions (evaluation benchmark, interesting qualitative analysis) more clear, and highlighted the potential future work in the third point in the conclusion.

(continued in reply)

Thank you again Reviewer ZAuw for your thoughtful feedback. We have attempted to address your concerns in our rebuttal and in our latest revision of the paper. We would appreciate it if you could look over our rebuttal and update your score if it has addressed your concerns. We're also happy to address any additional concerns you might have before the end of the discussion period on November 22 (tomorrow).

Thank you for your review! We're glad that you agree that benchmarking resistance to prompt injection is a valuable contribution and that you found our work novel. We also found your feedback in the weaknesses section particularly helpful, and have updated the paper in response. We've uploaded a new copy of the paper (please see the shared top-level message for details), and we address your specific points in more detail below.

Weakness 1

The paper's setting is initially hard to grasp; authors should aim to explain the threat model clearly, using notations and specifying the level of access attackers have.

Our new revision uses more precise notation in the introduction, and additional detail on the attacker and defenders' abilities and goals. The relevant paragraphs now read as follows:

The Tensor Trust web game simulates a bank. Each player has a balance, which they can increase by either coming up with successful attacks or by creating a defense that rebuffs attacks. In this section, we will explain the mechanics of defending and attacking, as well as implementation details on how we evaluate the attacks and defenses submitted by users.

Notation
We use $\mathcal V$ to denote a token vocabulary and $L : \mathcal V^* \times \mathcal V^* \times \mathcal V^* \to \mathcal{V}^*$ to denote an LLM that takes in three strings and outputs a single response string. $G : \mathcal V^* \to \{T, F\}$ denotes a goal predicate that determines whether a string says "access granted" (achieved using the regex in Appendix B).

Defending
Each account has a defense which consists of three prompts: an opening defense $d_{\text{open}}$, an access code $c_{\text{access}}$, and a closing defense $d_{\text{close}}$, as shown in Fig. 2. When a user saves a defense, we validate it by sandwiching their access code between the opening and closing defense and feeding it to the LLM $L$. The access code can only be saved if it makes the LLM output "access granted". In other words, $G(L(d_{\text{open}}, c_{\text{access}}, d_{\text{close}}))$ must be true.

Attacking
A player can select any other player's account and submit an attack against it. The text of the first player's attack, $c_{\text{attack}}$, is sandwiched between the defending player's opening and closing defense ($d_{\text{open}}$ and $d_{\text{close}}$), and then fed into the LLM $L$. If the LLM outputs "access granted" (i.e., $G(L(d_{\text{open}}, c_{\text{attack}}, d_{\text{close}}))$ is true), the attacker steals a fraction of the defender's money. Otherwise, the defender is granted a small amount of money for rebuffing the attack. The attacker cannot see $d_{\text{open}}$ or $d_{\text{close}}$, but can see the LLM's response to their attack. In the game, this is depicted as in Fig. 2.

Please let us know if we can make this more clear!

Weakness 2

The paper should establish a connection to Textual Backdoor attacks, even though these attacks typically require a more significant level of access to LLMs or their pretraining data than the setting the authors are primarily interested in. This additional context would help improve the clear understanding of the threat model in which Tensor Trust operates.

Thanks for the suggestion! We added the following paragraph to the related work section (section 7) to connect our work with past work on training-time attacks, including textual backdoor attacks. In this paragraph, Dai et al., Chen et al., Qi et al., and Wallace et al. belong to this category. If we missed any work that you think would be useful to add to the paper, we would be happy to add them.

Other past work considers training-time attacks. This might include poisoning a model’s training set with samples that cause it to misclassify certain inputs at test time (Biggio et al., 2012; Dai et al., 2019; Chen et al., 2021; Qi et al., 2021; Wallace et al., 2020), or fine-tuning an LLM to remove safety features (Qi et al., 2023). These papers all assume that the attacker has some degree of control over the training process (e.g. the ability to corrupt a small fraction of the training set). In contrast, we consider only test-time attacks on LLMs that have already been trained.

Please consider updating your score if these changes have adequately addressed your concerns about the paper, and let us know if there’s any other way that we can improve the writing.

Thanks for the modifications. I have increased my score.