Effective Prompt Extraction from Language Models

R2. System prompt definition is not clear

We consider system prompts as something set by the LLM service provider (e.g., “You are a helpful and harmless chatbot”), and the user prompt typically involves instructions on performing a task (e.g., “Translate everything I say to Spanish.”). Main results in Table 5 are actually on extracting these instruction-style user prompts. The main reason we don’t do this for actual system prompts is that there aren’t many LLM systems with system prompts available. We can indeed try using a standard system prompt, but we don’t believe it would make a meaningful difference to the experiment results.

We would also like to point out that in Section 6, we try our prompt extraction method on a few LLM services to extract their actual system prompts and observe seemingly successful results, but we can’t compute success metrics since we don’t have access to groundtruth system prompts of these services.

R3. Defense by paraphrasing

Indeed, exact prompt extraction can be defended through paraphrasing the output of language models, but the paraphrased output likely still reveals the same private information, and can work similarly well when used as instructions. We believe that the simplicity of our natural-language-based attack is in fact a strength: people can just take the set of attack queries and see if they compromise secrecy of prompts by running generation.

Q1.

Results in Table 1 are averaged (binarized) successes over the test sets. Are there specific places where we reported real-valued metrics and it’s not clear what they mean?

Q2.

First, we have a set of Dev prompts that are not used for final evaluation. We run prompt extraction over this dataset for all models (e.g., GPT-4 and Llama-2) to create 16,000 prompt, extraction pairs. Since we have access to both prompts and extractions, we can compute groundtruth token leakage (specifically, using the Rouge-L recall metric). The DeBERTa model is fine-tuned on extractions to predict token leakage. We will clarify in the final paper. We would also like to point out that code, dataset and model weights are already open-sourced.

Q3.

Permutations are always randomly generated. What this means is for a set of extractions generated by different attack queries, we randomly permute the extractions, feed them into the DeBERTa model to predict token leakage. We will clarify this in the paper.

Q4.

Please see our previous response on system prompt definition.

Method lacks novelty

We don’t have access to the paper you mentioned, and we assume you are referring to the “Language Model Inversion” (LMI) paper, which wasn’t public at the time of writing. LMI and our work both extract prompts, but there are a few key differences:

• Different assumptions of what an adversary can access: our attack assumes black-box access to a generation API, and LMI requires logit access. Logits can be recovered from a black-box API, but recovery of the full distribution requires $\mathcal{O}(\mathcal{V})$ number of queries. This means LMI cannot be cheaply applied to a black-box generation interface.
• Our attack is purely natural language, and doesn’t require a separate inversion model.

We will cite the LMI paper and incorporate relevant discussions.

Why DeBERTa?

Each attack query produces a different response, and we need a method to distinguish good extractions from clearly bad ones, and a language model can be trained to do this effectively. The model outputs a score for each extraction and we use the one with the highest score to run evaluations. If you are comfortable with having multiple extractions, the DeBERTa model is not necessary. It is simply a way of choosing a promising extraction out of all the candidates.

Why 105 attack queries?

There are 105 attack queries because the authors wrote a set of 5 queries and queried GPT-4 to generate an additional 100 queries. In preliminary experiments, we find just the 5 handwritten queries are already effective. We will add an additional analysis of attack success vs. # of queries.

Test set in the scale of a few hundred examples does not show effectiveness of method

We disagree with the reviewer. Even on our smallest test set out of 3, the 95% confidence intervals of our exact- and approx-match results are bounded above by $\pm$7.9%. We will report confidence intervals where relevant.

No systematic evaluation of system prompt extraction

We acknowledge that Table 1 does not show results of system-level, but user-level prompt extraction, but we want to highlight that such an evaluation is impossible: at the time of writing, there were only a small number of foundational LLMs, and their system prompts are generally secret.

Our extraction experiments on 4 system-level chatbots point to successful extraction, but it is impossible to be certain. We will qualify our claims where appropriate to make it clear that our main evaluations are done on user-level prompts.

Thanks for your positive review of our work. To address your question, we have only tried fine-tuning DeBERTa. We chose DeBERTa because it is one of the strongest masked language models, which can naturally be used for the task of predicting token leakage. Ensembling could improve predictive performance further, but given that the precision is already high, gain from ensembling would be limited.

Concerns about the correctness

The main concern of the reviewer is that we report prompt extraction success results based on an approximate string matching metric. To clarify, the approx-match metric does not consider an extraction successful when the majority of tokens are leaked; it considers an extraction successful when >90% of tokens are leaked (see its precise definition in Section 2, “metrics of success”). We also report extraction successes under an exact-match metric, which only considers an extraction successful if the entire prompt is leaked, and exact-match success rates remain high.

The reviewer’s confusion may stem from how we phrased our result in a few places (e.g.., “The majority of prompts can be extracted across heldout datasets and models.” in Table 1 caption). When we say “the majority of prompts can be extracted”, we always mean “more than 50% of prompts in the dataset can be extracted,” rather than “more than 50% of tokens in prompts can be extracted.” We will make the language more clear in the next version.

More discussion on how different types of prompts have very different rate of being extracted

We will add qualitative examples from all three sources of prompts and provide a discussion on what might have made prompt extraction more difficult.