SPIN: Self-Supervised Prompt INjection

Thank you for taking the time to read through our paper and giving feedback. We are glad that the reviewer finds our paper to be novel and effective. We address the questions and suggestions below:

1. Types of Attacks. Our defense is able to cover both jailbreak and prompt injection attacks. We revised our paper to follow your suggestions within the abstract and figure 1, and include the defenses mentioned within the related work. Below we also include a simple comparison to StruQ [1]:

1. TPR and FPR. Thank you for mentioning the FPR and TPR curves, we have added a discussion of them in the results section. In the interject task, the FPR is calculated by assuming malicious requests without attacks as positive. In actual experimentation, the alignment will actually prevent the malicious requests without attacks from generating the ‘Paris’ token, leading to a lower FPR than shown in the ROC curves.

2. Computational Costs. Thank you for raising questions about reasoning costs. We agree that our method introduces an extra reasoning cost to the conversation. However, for critical applications where robustness is important, it is not desirable to make the wrong prediction just to make it faster, and our framework can provide safety. Since our latency is in seconds, our algorithm is still realistic to implement, as Interjection and Repeat add less than 2 seconds to the response. In addition, since our method is easily toggle-able, we will only run the defense algorithm when the task is important, and we can choose how many steps to update at inference time, which can significantly accelerate our method.

3. Additional Models and Datasets. Thank you for your suggestion on including more model experiments and datasets. Below we demonstrate our defense method on 2 additional models, and our model is still able to be effective. We also showcase results on 2 additional benign datasets to show that our defense method is still able to maintain performance on normal use cases.

   Performance on Additional Models with 100 samples

   Model	JailbreakChat	ICA	Adv. Instr
   Llama 3.1 (No Defense)	70%	19%	60%
   Llama 3.1 + SPIN	1%	1%	0%
   Mistral-7B (No defense)	100%	74%	92%
   Mistral-7B + SPIN	0%	2%	0%

Performance on Benign datasets with 100 samples (Results are based on actual token generated, rather than logits)

[1] StruQ: Defending Against Prompt Injection with Structured Queries

Thank you for taking the time to read through our paper and giving feedback. We are glad that the reviewer finds our paper to be novel and effective. We address the questions and suggestions below:

1. Figures and Captions. Thank you for catching that typo. We originally meant to refer to the Blue box which represents our defense method and we have made that clear now in our revision.
2. Choice of Detection and Thresholds. Thank you for your inputs on the insights. We chose the two detection tasks because they had objective ways to measure the correctness, and were easily answerable on benign prompts for most models. Our approach is also general enough that users can design their own questions within the interject task. In addition to our ablation studies in figure 7, below we also include a few ablation on different prompts.

Choice of Prompts on Vicuna

1. Related Work on Existing Defenses. We have made a new paragraph in Related work called Defense Methods which furthers work relating to alignment, and discusses more defense methods, including a few that we compare our own method with.
2. Additional Models. Thank you for the suggestions about additional model experiments. Following your suggestion, we included 2 new models to demonstrate our defense method’s broader applicability. Our defense method is effective across these different models.

Performance on Additional Models with 100 samples

1. Defense Suffix. In our ablation studies we found that appending our tokens at the front or end had similar results, and we chose to append it in the front to clearly differentiate from existing GCG attacks. This is a design choice and will work just as well at the end. Thank you also for catching the threshold typo, and we have revised it in the updated version.
2. Practical Use Case. We find your question on practical use cases to be important, and since our ‘repeat’ and ‘interjection’ tasks are for detection, the final output generated will be affected on benign prompts. On TriviaQA performance we showed that our defense method is able to maintain similar performance, and below we include two more datasets to show real-world practicality

Performance on Benign datasets with 100 samples (Results are based on token generation, rather than max logits)

1. Citation. Thank you for bringing this to our attention, we have now properly added Jain et al’s paper to our bibliography

Thank you for taking the time to read through our paper and giving feedback. We agree with the reviewer that our method is able to adaptively defend against malicious attacks without the upfront training time costs. Below, we address the questions and suggestions:

1. LLM Generalization + Llama Guard. Thank you for the suggestions of additional models. Following your suggestions, we have included experiments for two additional models below. In both cases, our defense method is still effectively in preventing jailbreak attacks. In addition we have also included comparisons to Llama Guard [1] as you suggested.

Performance on Additional Models with 100 samples

1. Threshold Calculations. For our defense, we agree it is valuable to have a method that is able to generalize across different models and datasets. The same threshold seen for the repeat in Figure 4 is used for both models, and all the datasets that we use. It was calculated by finding the loss for each prompt in a benign and malicious attacks database, then automatically finding the best threshold to separate the classes. This is a threshold that still has great performance on models different from the one it was calculated on. As for Interjection, we can also check whether it generates the correct token ‘Paris’ within the first 10 tokens. This allows the detection to work across models in a more flexible way.
2. Domains. Our framework is general enough to adapt to prompts from specialized domains. We also agree that domain specific prompts can be useful to further improve the domain, and in addition, SPIN allows the user to customize the defense injections to account for this specialization. The interjection task can be changed to instead ask a more domain-specific question, which will test for specialized use cases.

[1] Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, and Madian Khabsa. Llama guard: Llm- based input-output safeguard for human-ai conversations, 2023.

Thank you for all the helpful feedback, we will emphasize the discussion on thresholds and generalization in the revisions. Following your suggestion for an ensemble of benign and malicious data, below is a table of results across different benign and malicious datasets and we show that the thresholding is consistent.

Thresholding on ensemble datasets for Llama-2

Accuracy using Different Thresholds:

Levenshtein Distance Distribution for various datasets

Thank you for taking the time to read through our paper and giving feedback. We are glad that the reviewer finds our paper to be a novel idea way for prompt injections to enhance the model’s safety. We address the questions and suggestions below:

1. Title. Thank you for bringing up the title. We added to the title ‘D-SPIN: Defensive Self-supervised Prompt INjections’ to clarify that we mean to introduce a defense method.
2. Inference Runtime. In Table 1, we list out only the cost of doing the check by itself. We will make sure this is more clear in the table. In the case of interjection, since we only look at the initial 10 tokens, we do have to wait as long compared to standard inference. The same is true for repeat because these language models are so verbose that repeating the question is usually shorter than the answering the question.
3. Language Models. Thank you for your question on different language models. In our paper we tested on various models with Llama-2 already being more secure against prompt injections. Even in the stricter model, because our defense method adds on to the alignment, it only improves on the safety of the base model. We also include a few brief experiments on Llama-3 below to show that a secure model can still be improved by our defense:

Performance on Additional Model with 100 samples