Safeguarding System Prompts for LLMs

Dear reviewer,

Thank you for your valuable and constructive feedback. We list below our replies to your concerns.

(Weakness 1, 4, and Question 2): The definition of prompt confidentiality makes no sense to me. Mutual information between the LLM response and the prompt zero only if they are statistically independent. A system prompt that has zero statistical influence on responses is useless. Simply throw away the system prompt, no need to worry about its confidentiality. || The word "privacy" is completely misused to mean "confidentiality". || There is zero mutual information between the responses and the prompt.

To clarify, PromptKeeper is not designed to achieve confidentiality of the system prompt—meaning zero mutual information—between the system prompt and responses. Instead, it is intended to help navigate a practical tradeoff between preserving prompt privacy and maintaining prompt effectiveness.

As motivated in Sec. 3, the challenge of protecting system prompts lies in quantifying and mitigating partial leakage within natural-language responses, a problem for which no widely accepted metrics currently exist. PromptKeeper addresses this gap by leveraging hypothesis testing as a statistical framework, rather than relying on subjective or potentially flawed artificial metrics. In this framework, zero leakage serves as a conceptual baseline for evaluating leakage, not as the target outcome. The degree of tolerable leakage—i.e., "how far a tolerable leakage can be from zero leakage"—is configurable through $\alpha$, the target false positive rate. This configurability implies that PromptKeeper does not default to ignoring the system prompt.

As further illustrated in Figs. 4, 6, and 7, setting $\alpha = 0$ enforces the strictest defense, resembling a no-prompt scenario. Conversely, setting $\alpha = 1$ results in a vacuous defense, approximating a no-defense scenario. Between these extremes, increasing $\alpha$ allows for greater prompt effectiveness at the cost of reduced privacy. This flexible design enables service providers to navigate the trade-off between privacy and adherence according to their specific requirements. We will revise the paper's narrative to better articulate these design rationales.

Regarding terminology, we use the term "privacy" to encompass the broader notion of information protection, which includes controlling partial information leakage. In contrast, "confidentiality" implies zero leakage, a stricter notion that does not align with PromptKeeper’s objectives.

(Weakness 2 and Question 1): Evaluation of the proposed defense claims to show that responses generated without the prompt have approximately the same quality as responses generated with the prompt. To me, this is not evidence that the defense works. This is evidence that system prompts used for evaluation are useless because they do not improve quality of responses. The correct baseline is to generate all responses without the system prompt and measure their quality vs. responses based on the system prompt. || Quality of responses with the prompt is significantly higher than without (i.e., the prompt is useful).

As introduced in Sec. 5.3, we did include the two recommended baselines: one with no prompt and another with no defense. This baseline comparison serves as a benchmark to evaluate our tradeoff between prompt privacy and prompt effectiveness.

As mentioned above, PromptKeeper is designed as a flexible defense rather than a point solution, with an adjustable parameter, $\alpha$, allowing users to balance between privacy and response quality. Consequently, the evaluated response quality under PromptKeeper varies with different $\alpha$ values rather than staying constant. As Fig. 4, 6, and 7 illustrate, when $\alpha$ is set to a low value, high defense effectiveness is achieved, though the response quality may approach that of the "no prompt" baseline. Conversely, as $\alpha$ increases, defense effectiveness decreases, and response quality aligns more closely with the "no defense" baseline.

This flexibility enables users to select an $\alpha$ value that best suits their privacy and quality needs. We hope this clarifies our evaluation approach, but please feel free to let us know if you have further questions regarding the results or methodology.

Dear reviewer,

Thank you for your valuable and constructive feedback. We list below our replies to your concerns.

(Weakness 1): The formatting needs improvement, with excessive whitespace in several areas, particularly on page 7.

In the revised manuscript, we have eliminated unnecessary whitespace by enabling line breaks in lengthy references (e.g., the name text-embeddings-ada-002).

(Weakness 2): The data in the tables require alignment for better aesthetics.

In the revised paper, we have improved the formatting of all tables by aligning the data within each column based on the plus-minus signs to enhance readability and aesthetics.

(Weakness 3)：PromptKeeper necessitates full knowledge of the service provider's system prompt, raising questions about its applicability in today’s API landscape, such as with GPT stores.

We emphasize that PromptKeeper is designed to be deployed by the service provider and that the assumption of full access to the system prompt aligns with standard practice. For instance, on popular LLM platforms like OpenAI’s GPT Store and Poe, developers are required to specify their system prompts in plaintext, making the service provider the effective prompt owner.

Moreover, this practice is expected to persist in the near future because, without access to the system prompt, user query processing would need to rely on trusted execution environments (TEEs) or secure multiparty computation (SMPC), both of which are currently impractical due to their high computational and financial costs. Until these techniques become feasible for industrial deployment, LLM deployment will continue in plaintext, allowing service providers to retain access to the system prompt.

(Question 1): How is PromptKeeper's impact on benign prompts measured?

As described in Sec. 5.4, we developed an LLM-based methodology to evaluate PromptKeeper's impact on benign user queries, focusing on chat quality. As discussed in the last two paragraphs of Sec. 6.1 and illustrated in Fig. 4, 6, and 7, while PromptKeeper may slightly reduce chat quality to protect system prompts, it achieves higher chat quality at comparable defense effectiveness relative to alternative defense mechanisms.

Additionally, PromptKeeper’s configurable parameter, $\alpha$ (target false positive rate), allows for fine-grained control, enabling a flexible balance between chat quality and defense effectiveness.

(Question 2): Is there a risk of false negatives?

As defined in Sec. 3, a false negative occurs when the test incorrectly indicates non-zero leakage despite zero leakage. Like any binary classification system, PromptKeeper carries a risk of false negatives. Our leakage identification method offers flexibility in managing the trade-off between the actual false negative rate and actual false positive rate, where the latter is proportionate with $\alpha$, i.e., the target false positive rate. Theoretically, increasing $\alpha$ reduces the empirical false negative rate, allowing service providers to select an $\alpha$ value that best aligns with their specific requirements.

However, it is important to emphasize that the actual false negative rate is intractable in practice, as constructing and validating positive cases—responses generated with a system prompt that leak no information about the prompt—is challenging. Therefore, we can only discuss the risk of false negatives conceptually, without introducing new empirical evaluations.

Dear reviewer,

Thank you for your valuable and constructive feedback. We list below our replies to your concerns.

(Weakness 1): Major conceptual flaw: Defense just ignores the system prompt. The proposed defense aims to achieve zero MI between generations and the system prompt. This happens if and only if the system prompt is independent of the response. Hence, the ideal instantiation of this concept is to simply drop the system prompt. This also manifests in the instantiation of the proposed defense, which effectively test if the model adheres to the system prompt and regenerates the response without a system prompt if yes. However, any useful defense should minimize leakage of information in the system prompt while retaining a minimum level of adherence to the system prompt; otherwise, simply dropping the system prompt is a more straightforward defense. I believe this conceptual flaw requires substantial changes to the setup and instantiation of this paper's proposed defense. (An example of this issue is in Sec. 6.2: The chat model can either adhere to the 100-word limit, or protect this information, but not both simultaneously.)

To clarify, PromptKeeper is not designed to achieve zero leakage of the system prompt. Instead, it indeed minimizes leakage while retaining essential adherence to the prompt.

As motivated in Sec. 3, the challenge of protecting system prompts lies in quantifying and mitigating partial leakage within natural-language responses, a problem for which no widely accepted metrics currently exist. PromptKeeper addresses this gap by leveraging hypothesis testing as a statistical framework, rather than relying on subjective or potentially flawed artificial metrics. In this framework, zero leakage serves as a conceptual baseline for evaluating leakage, not as the target outcome. The degree of tolerable leakage—i.e., "how far a tolerable leakage can be from zero leakage"—is configurable through $\alpha$, the target false positive rate. This configurability implies that PromptKeeper does not default to ignoring the system prompt.

As further illustrated in Figs. 4, 6, and 7, setting $\alpha = 0$ enforces the strictest defense, resembling a no-prompt scenario. Conversely, setting $\alpha = 1$ results in a vacuous defense, approximating a no-defense scenario. Between these extremes, increasing $\alpha$ allows for greater prompt effectiveness at the cost of reduced privacy. This flexible design enables service providers to navigate the trade-off between privacy and adherence according to their specific requirements. We will revise the paper's narrative to better articulate these design rationales.

(Weakness 2): The evaluation measures response quality and adherence to the system prompt in a combined metric, but should use two separate metrics. Adherence to the system prompt is the most important metric and can be measured more easily than quality, but could be anti-correlated with quality. This could also explain the relatively small differences on the x-axis in Figure 4 ab.

We agree that prompt adherence is a core metric for evaluating the impact of PromptKeeper, particularly given its on-demand regeneration design. Following your suggestion, we have separated prompt adherence from response quality and now report adherence as a standalone metric in the revised paper. As you anticipated, this separation reveals larger differences along the x-axes in Figs. 4, 6, and 7, making the tradeoff between prompt privacy and prompt adherence more apparent.

It is also noteworthy that these updated experiments confirm that all of our original conclusions remain valid. Specifically, PromptKeeper effectively balances defense effectiveness with prompt adherence and provides a comprehensive, fine-grained mechanism for navigating the tradeoff within the identified sweet spots.

After the initial review, I noticed that the method section of this paper may have plagiarized Carlini et al., 2022 on membership inference attack. Key ingredients of the method including the hypothesis testing framework, derivation of the test objective, and fitting Gaussians seem to be exactly the same as that paper, just used in a different context. Chairs/ethical reviewers should verify this case.

Authors of the paper: please clarify the situation as well.

First, apologies for posting this comment on the wrong thread. I meant to post it under my review. Some comments:

Like many of these peer-reviewed works, we applied and adapted established mathematical tools to address a novel context in our research. All derivations, test objectives, and applications of these frameworks in our work are original and specifically tailored to the problem we address.

I don't find your justification here reasonable. We all use prior methods and tools to do research, but you are plagarizing by definition if you do so without attribution, and sell it as if this was completely new. I am intrigued by the reason why the authors felt the need to edit your methods section by renaming variables and changing equations, if you truly believe all your work is original.

Context: the authors have updated the PDF at some point during the response period. The original submission provides a much clearer picture of this case. I am not sure if the original submission is still visible on OpenReview, but I have the original version in case someone needs access.

1. Lack of Engagement with Our Responses

I am engaging with you in good faith. Your responses were posted 7 hours ago, and I need some to address your responses.

2. Misunderstandings and Biases in Feedback:

I could have misunderstood the contribution of you paper, but plagarism is an entirely different issue. Please do not use an ad hominem attack on me as a distraction from the actual issue here.

Dear reviewer,

Thank you for your valuable and constructive feedback. We list below our replies to your concerns.

(Weakness 1a): The method to "robustly" identify prompt leakage has a few key flaws. The authors started by motivating "detecting prompt leakage as hypothesis testing on prompt and response has 0 mutual information". In reality, any system prompt surely biases the distribution of responses in some way (otherwise it's completely useless)! Then, observing the response must reduce some uncertainty in the system prompt (i.e., mutual information > 0). So the proposed test is vacuous: the null hypothesis should always be true.

To clarify, PromptKeeper is not designed to achieve zero mutual information between the system prompt and the generated response. Instead, its goal is to minimize the leakage of the prompt while retaining its necessary effectiveness in guiding responses.

As motivated in Sec. 3, the challenge of protecting system prompts lies in quantifying and mitigating partial leakage within natural-language responses, a problem for which no widely accepted metrics currently exist. PromptKeeper addresses this gap by leveraging hypothesis testing as a statistical framework, rather than relying on subjective or potentially flawed artificial metrics. In this framework, zero leakage serves as a conceptual baseline for evaluating leakage, not as the target outcome. The degree of tolerable leakage—i.e., "how far a tolerable leakage can be from zero leakage"—is configurable through $\alpha$, the target false positive rate. This configurability implies that the proposed testing does not default to be vacuous.

As further illustrated in Figs. 4, 6, and 7, setting $\alpha = 0$ enforces the strictest defense, resembling a no-prompt scenario. Conversely, setting $\alpha = 1$ results in a vacuous defense, approximating a no-defense scenario. Between these extremes, increasing $\alpha$ allows for greater prompt effectiveness at the cost of reduced privacy. This flexible design enables service providers to navigate the trade-off between privacy and adherence according to their specific requirements. We will revise the paper's narrative to better articulate these design rationales.

(Weakness 1b, 1c, and Question 2): The use of technical language is confusing. For example $\mathbb{Q}$ is introduced as a distribution, defined as a set, and used like a random variable (e.g., $p\mid\mathbb{Q}(p, q)$) in text. In Eq. 4, what does it mean to compute the probability of mean log likelihood (a number, not an event) conditioned on $\mathbb{Q}$? It is not obvious to me you are basically computing the mean and variance of mean log likelihoods offline and computing tail probability of a gaussian until much later. || The actual method is pretty straightforward: computing log likelihoods of the provided response with and without prompt, and compute how "abnormal" it is. A high ratio means the response likely contains information about the prompt. Such a method fundamentally can't distinguish between the model making use of its system prompt, vs. the model leaking its system prompt! For example, consider a system prompt "Translate my instruction to Python code.", and a user query "What's tan(2pi)?" Without the system prompt, the model would assign very low probability to something like math.tan(2 * math.pi). So, it would get flagged by the method as likely prompt leakage. You could calibrate $\mathbb{Q}$ offline, but I am still not convinced that such a approach could tell apart the two cases generally. || "It is worth noting that obtaining the mean log-likelihood does not require extra computation." Don't you need an extra forward pass to compute $\mathbb{Q}_{\textrm{zero}}$?

The confusion regarding our hypothesis testing process (Sec. 3) likely arose from the earlier unclear notation. In the revised paper, we have thoroughly reorganized and clarified the presentation of our notation. This enables us to provide a clearer explanation of the process, which we also summarize briefly here:

1. For a response under evaluation, its mean log-likelihood is obtained as a by-product of the generation process.
2. Using the pre-computed distributions $\tilde{Q}^\ast_{\textrm{zero}}$ and $\tilde{Q}^\ast_{\textrm{other}}$, we compute the probability densities of the obtained mean log-likelihood value.
3. We compute the ratio of these densities to perform hypothesis testing.

As one can see, our hypothesis testing does not compare log-likelihoods of the provided response with and without prompt, as suggested in your example. What the hypothesis testing instead compares are the probability densities of the same mean log-likelihood value under the two pre-computed distributions $\tilde{Q}^\ast_{\textrm{zero}}$ and $\tilde{Q}^\ast_{\textrm{other}}$. This also confirms that no additional forward pass is required during the process.