Has My System Prompt Been Used? Large Language Model Prompt Membership Inference

7. [Comparison to prompt extraction baselines] We ran additional experiments comparing PLeak [4] – the most high performing of the existing prompt reconstruction approaches (referenced in other reviewers' comments) to Prompt Detective in the prompt membership setting. We used the optimal recommended setup for real-world chatbots from section 5.2 of the PLeak paper – we computed 4 Adversarial Queries with PLeak and LLAMA-2 13B as the shadow model as recommended, and we used ChatGPT-Roles as the shadow domain dataset to minimize domain shift for PLeak. We observed that PLeak sometimes recovers large parts of target prompts even when there is no exact substring match, and that using the edit distance below the threshold of 0.2 to find matches maximizes PLeak’s performance in the prompt membership inference setting. To further maximize the performance of the PLeak method, we also aggregate the reconstructions across the 4 Adversarial Queries (AQs) by taking the best reconstruction match (this aggregation approach is infeasible in prompt reconstruction setting where the target prompt is unknown but can be used to obtain best results in prompt membership inference setting where we know the reference prompt). We then applied these adversarial prompt extraction queries to LLAMA-2 13B as the target model with system prompts from Awesome-ChatGPT-Prompts and computed False Positive and False Negative rates for direct comparison with the results of Prompt Detective reported in Table 1 of our paper. We report the results below:

We see that Prompt Detective significantly outperforms PLeak in the prompt membership inference setting which is expected since Prompt Detective is specifically tailored to work in the verification setup while PLeak is geared towards solving a different problem of prompt reconstruction. We added these results to Appendix B.1.

8. [Black box setting] We respectfully disagree that the black box setting was overlooked. The paper includes experiments showcasing the effectiveness of Prompt Detective in a practical black box scenario and reports the results in Table 3.

9. [Highlighting the ability to separate even extremely similar system prompts] While we agree that related findings of LLM brittleness and sensitivity to prompts exist and are well-known, our work highlights a distinct finding – the ability to separate even extremely similar system prompts, like the ones differing only by a typo, as they manifest in distinct response distributions (by eye, responses to such two system prompts of extreme similarity would be indistinguishable). See Appendix C.2 for a case study on prompts differing by a typo. We also point out that other reviewers found this particular finding surprising and important, as mentioned by Reviewer L9Uj: “The paper has some surprising results, at line 448 ( system prompts that differ only by a typo as an example of extreme similarity (see Appendix C for details). I think this result is quite important and may have broader implication…”

In response to other reviewers feedback, we have also performed an ablation study on Prompt Detective embeddings and included ROC curves for Prompt Detective in the paper. Please, see our general response for details.

We sincerely thank you once again for your detailed feedback. We hope our responses have addressed your questions, and kindly ask you to consider increasing your score in light of the clarifications, additional experiments, and writing improvements we have provided.

References:

[1] https://prompti.ai/chatgpt-prompt/

[2] https://promptbase.com/

[3] https://github.com/jujumilk3/leaked-system-prompts/tree/main

[4] Hui, B., Yuan, H., Gong, N., Burlina, P. and Cao, Y., 2024. PLeak: Prompt Leaking Attacks against Large Language Model Applications. arXiv preprint arXiv:2405.06823.

[5] Zhang, Y., Carlini, N. and Ippolito, D., 2024, August. Effective prompt extraction from language models. In First Conference on Language Modeling.

Thank you for your review and feedback. We appreciate your recognition of our paper's clear presentation and positive results. We address your points below:

1. [Problem motivation and importance] Carefully crafted system prompts play a crucial role in shaping LLM outputs and driving performance in application domains, in fact some prompts are valuable enough to be sold at online marketplaces [1,2]. Separately, hidden chatbot system prompts sometimes contain instructions that the users may want to be aware of such as instructions on handling private information of the user [3]. Regarding prompt reconstruction, it is certainly possible and methods such as PLeak [4] (the most promising prompt reconstruction approach) achieve reconstruction in real-world chatbots relatively successfully (with 68% success rate). System prompts may also be leaked through means other than reconstruction methods [3] and utilized by adversaries in their chatbots. However, the prompt reconstruction problem is indeed challenging and distinct from the prompt membership inference and what we meant in introduction is that the success rate of prompt reconstruction methods (e.g., PLeak’s 68%) is not high enough to reliably verify prompt reuse and more effective methods tailored to this problem such as Prompt Detective can be developed. In fact, we ran additional experiments comparing Prompt Detective vs PLeak as a prompt reconstruction baseline in the prompt membership inference setting and found that Prompt Detective is significantly more effective (achieving 0.05 FNR and 0.00 FPR vs PLeak’s 0.46 FNR and 0.00 FPR). Please, refer to our response to your point 7. [Comparison to prompt extraction baselines] below for more details on this experiment (and also to our general response or Appendix B.1 of the paper).

2. [Related work] We appreciate this feedback and have revised our introduction and related work section to include a more clear discussion of prompt reconstruction methods to better position our contribution within the literature. Regarding the word "usually" in line 62, we removed that word and clarified that while some reconstruction methods provide confidence scores (namely, [5] which was already cited), they don't offer statistical guarantees for prompt usage verification. We have also added the comparison to PLeak (the most performant prompt reconstruction baseline) mentioned above to Appendix B.1.

3. [Methodology motivation] We revised Section 3.1 to clarify the motivating example.

4. [Novelty] While our method does rely on existing research (as most research does), its novelty lies in the application to the prompt membership inference problem and finding the right combination of components to develop an effective statistical framework for prompt reuse verification (more effective, for example, than utilizing prompt reconstruction methods for the same purpose as evidenced by the comparison experiment to PLeak). While some design choices may seem simple, they are in fact effective, as pointed out by Reviewer vCak: “Prompt Detective is a straightforward hypothesis test without any unnecessary moving parts, yet done properly such that it achieves good performance in terms of what it set out to do“.

5. [Task prompts] Regarding task prompts, as part of finding the effective recipe for the statistical framework, we for example explore the optimal allocation of a fixed budget of generations in Figure 4 – we find that having a larger number of shorter responses and diversified sets of different task prompts which elicit responses that are directly influenced by system prompts is most useful for effective separation. (The best particular setup in the Figure 4 experiment is to use 10 task prompts with 50 generations, each 64-token-long).

6. [Tasks in section 5.1, false negative rates in Table 2] While some of the system prompts in our general experiments are fairly distinct, these datasets also include system prompts for similar roles which are more challenging to separate in negative cases (where the known system prompt differs from the system prompt used by the model). Additionally, the general experiments also confirm the ability of Prompt Detective to reliably verify positive cases. Section 5.2 is devoted to the exploration of much more challenging negative cases – hard examples. False negative rates are not included in Table 2 because there are no positive cases – the experiments are focused on separating the original system prompt from other, distinct system prompts of varying similarity to the original.

Dear Reviewer hT6p,

Thank you for your time and effort in reviewing our paper and rebuttal.

We have added new experiments and results to address your concerns in the updated draft.

We hope our responses have addressed your questions, and kindly ask you to consider increasing your score in light of the clarifications, additional experiments, and results we have provided. Please let us know if you have further questions as the extended discussion period comes to an end. Thank you!

Dear Reviewer hT6p,

Thank you again for your feedback and time!

As the extended discussion period comes to an end tomorrow, we wanted to post a gentle reminder that we added new experiments and results to address your concerns. Please see our responses to your review as well as the general response. We hope our responses address your questions, and kindly ask you to consider increasing your score in light of the clarifications, additional experiments, and results we provided.

Thank you for your thoughtful review and feedback. We are glad that you found our results on a typo case study surprising and important, and we address your comments and questions below:

1. [I think verifying if the prompt used is the same might not be needed.]

We argue that while prompt reconstruction methods attempt to deduce the content of a system prompt, they are prone to errors, often require white-box access to the model, and lack a reliable way to confirm whether the reconstructed prompt matches the actual prompt used. This is where Prompt Detective provides a distinct advantage—it offers a statistically significant verification mechanism that complements, rather than replaces, reconstruction methods. Even if reconstruction reveals an approximate prompt, it cannot provide a certifiable guarantee that the reconstructed prompt was actually employed by the system. Prompt Detective addresses this gap by offering an independent verification mechanism based on observable response distributions.

Despite these differences between the prompt reconstruction and prompt verification approaches, we ran additional experiments comparing PLeak [1] – the most high performing of the existing prompt reconstruction approaches [1-4] referenced in your and reviewer DuyU’s comments (per the results reported in each of the papers) to Prompt Detective in the prompt membership setting. We used the optimal recommended setup for real-world chatbots from section 5.2 of the PLeak paper – we computed 4 Adversarial Queries with PLeak and LLAMA-2 13B as the shadow model as recommended, and we used ChatGPT-Roles as the shadow domain dataset to minimize domain shift for PLeak. We observed that PLeak sometimes recovers large parts of target prompts even when there is no exact substring match, and that using the edit distance below the threshold of 0.2 to find matches maximizes PLeak’s performance in the prompt membership inference setting. To further maximize the performance of the PLeak method, we also aggregate the reconstructions across the 4 Adversarial Queries (AQs) by taking the best reconstruction match (this aggregation approach is infeasible in prompt reconstruction setting where the target prompt is unknown but can be used to obtain best results in prompt membership inference setting where we know the reference prompt). We then applied these adversarial prompt extraction queries to LLAMA-2 13B as the target model with system prompts from Awesome-ChatGPT-Prompts and computed False Positive and False Negative rates for direct comparison with the results of Prompt Detective reported in Table 1 of our paper.

We report the results below:

We see that Prompt Detective significantly outperforms PLeak in the prompt membership inference setting which is expected since Prompt Detective is specifically tailored to work in the verification setup while PLeak is geared towards solving a different problem of prompt reconstruction. We added these results to Appendix B.1.

2. [Revealing system prompts have broader implications.] While the main experiments in the paper focus on role-playing prompts, we have also demonstrated the generalizability of Prompt Detective to realistic, production-grade prompts. Specifically, in the typo case study, we examined the widely used Llama system prompt, which instructs the model to function as a helpful and harmless assistant. Our results show that Prompt Detective effectively distinguishes between two distinct general-purpose prompt variations, highlighting the robustness and applicability of our method to real-world scenarios.

3. [ Regarding the above point, it seems that the test is costly in terms of the number of queries needed.] As outlined in Section 4, our experiments use 50 to 500 generations per system prompt, depending on the level of similarity. When the prompt in the third-party language model closely resembles the proprietary one, up to 500 generations may be required to statistically detect differences. However, in most cases, Prompt Detective achieves reliable results with just 50 generations.

4. [Figure 2] In Figure 2 we illustrate examples of “hard prompts”, which are not the same as the proprietary prompts (so, negative cases), but which we assume would be harder to differentiate from the proprietary prompt because of the high similarity between them.

Dear Reviewer L9Uj,

We sincerely appreciate your efforts in reviewing and reading our paper and rebuttal. We have made a significant effort and added new experiments and results to address your concerns in the updated draft.

We kindly hope you will consider raising your score if our revisions and rebuttal have resolved your concerns. Please let us know if you have any further questions. Thanks again for your contribution to our paper.

• 5.5 [Task probes.] The task probe construction is automated using the Claude 3 Sonnet language model, please see Table 6 in Appendix for the prompts used to construct the task probes. Conceptually, the most important aspect for the task probe construction is to build task probes which elicit responses that are related to the system prompt and are directly influenced by it, otherwise the utility of the target LLM responses would be limited. The other important aspect is practical generation budget allocation for the task prompts – that is explored in ablation presented in Figure 4 and mentioned above.

• 5.6 [Ablation on embedding methods.] Prompted by your review, we ran additional experiments for an ablation on embedding methods. The results are included in Appendix B Table 4. In short, we observe no significant difference between performance of different encoder models when used in Prompt Detective. We experimented with gte-Qwen2-1.5B-instruct, jina-embeddings-v3 and mxbai-embed-large-v1 models from MTEB leaderboard [5] and found that our results are consistent across various embedding techniques. Originally, we simply chose BERT for its simplicity, wide adoption, and computational efficiency.

• 5.7 [Task prompt set in general experiments.] Having responses generated for the same collection of task prompts across different system prompts simplifies random sampling of negative cases – where the known system prompt differs from the system prompt used by the model.

6. [Black box setting.] We note that our black box setting is practical because, as we mention is Section 3.1 of the paper, prompt engineering is a much less resource-intensive task than developing or fine-tuning a custom language model, therefore, it is reasonable to assume that such chat bots which reuse system prompts are based on one of the publicly available language models such as API-based GPT models, Claude models, or open source models like Llama, Mistral, or others. For the black box experiments, for the pool of models we simply used all models we used in our general experiments, however, our method is similarly applicable to larger pools of models as well. We do agree that this setting can be considered a “dark gray” box, we believe that it is a practical setting nonetheless.

7. [Minor comments.] Thank you for your points, we added clarifications to the paper and added the citations to the related work section. The prompt used for constructing hard examples is already included in Table 6 in Appendix F. We have included a new section with a detailed explanation of each step in the algorithm in Appendix D. Regarding your question 4, since the null hypothesis corresponds to two distributions being the same, Type 1 error indicates the probability of rejecting the null hypothesis of equal distributions.

8. [Task probe construction prompt.] Please, see point 5.5.

We sincerely thank you once again for your detailed feedback. We hope our responses have addressed your questions, and kindly ask you to consider increasing your score in light of the clarifications, additional experiments, and writing improvements we have provided.

References:

[1] Zhang, Y., Carlini, N. and Ippolito, D., 2024, August. Effective prompt extraction from language models. In First Conference on Language Modeling.

[2] Hui, B., Yuan, H., Gong, N., Burlina, P. and Cao, Y., 2024. PLeak: Prompt Leaking Attacks against Large Language Model Applications. arXiv preprint arXiv:2405.06823.

[3] Geiping, J., Stein, A., Shu, M., Saifullah, K., Wen, Y. and Goldstein, T., Coercing llms to do and reveal (almost) anything, 2024. URL https://arxiv. org/abs/2402.14020.

[4] Zou, A., Wang, Z., Carlini, N., Nasr, M., Kolter, J.Z. and Fredrikson, M., 2023. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043.

[5] Muennighoff, N., Tazi, N., Magne, L. and Reimers, N., 2022. MTEB: Massive text embedding benchmark. arXiv preprint arXiv:2210.07316.

4. [Comparison to prompt extraction baselines.]

Prompted by your review, we ran additional experiments comparing PLeak [2] – the most high performing of the existing prompt reconstruction approaches (referenced in your and reviewer L9Uj’s comments) to Prompt Detective in the prompt membership setting. We used the optimal recommended setup for real-world chatbots from section 5.2 of the PLeak paper – we computed 4 Adversarial Queries with PLeak and LLAMA-2 13B as the shadow model as recommended, and we used ChatGPT-Roles as the shadow domain dataset to minimize domain shift for PLeak. We observed that PLeak sometimes recovers large parts of target prompts even when there is no exact substring match, and that using the edit distance below the threshold of 0.2 to find matches maximizes PLeak’s performance in the prompt membership inference setting. To further maximize the performance of the PLeak method, we also aggregate the reconstructions across the 4 Adversarial Queries (AQs) by taking the best reconstruction match (this aggregation approach is infeasible in prompt reconstruction setting where the target prompt is unknown but can be used to obtain best results in prompt membership inference setting where we know the reference prompt). We then applied these adversarial prompt extraction queries to LLAMA-2 13B as the target model with system prompts from Awesome-ChatGPT-Prompts and computed False Positive and False Negative rates for direct comparison with the results of Prompt Detective reported in Table 1 of our paper. We report the results below:

We see that Prompt Detective significantly outperforms PLeak in the prompt membership inference setting which is expected since Prompt Detective is specifically tailored to work in the verification setup while PLeak is geared towards solving a different problem of prompt reconstruction. We added these results to Appendix B.1.

5. [Experiment points.]

• 5.1 & 5.2 & 5.4 [Model families and sizes used in the General experiments and Hard Examples.] In our general experiments in Table 1, we report Prompt Detective performance across a variety of language model families and sizes – including both larger and smaller models, multiple models of the various open source families, and closed-source models. We believe that Table 1 clearly showcases the effectiveness of Prompt Detective with minor variations in performance across these settings, therefore we believe including more combinations of model sizes and families would result in suboptimal compute spend for a limited quantity of new research findings. The Hard Example section is a deep dive section where we explored the setting of highly similar system prompts and, following the similar logic of responsible use of compute resources and minimizing the carbon footprint, we decided to focus on the efficient variants of models powering popular real-world chatbots (as our results are fairly robust across model sizes and families experimentally and conceptually we would also expect that, we opted for the efficient Claude Haiku and GPT 3.5 vs larger and more expensive Claude or GPT variants). Prompted by your questions, we added a clarification on the models we used in our experiments to Appendix G.

• 5.3 [Figure 4.] Thank you for your questions. Figure 4 explores the optimal setup for a fixed budget of generations – on the left panel, it explores whether for the same number of LLM queries it is better to have more task prompts versus more responses per task prompt. While the trends are overall fairly similar, the diversified approach – with a larger number of task prompts yields more consistent results. The right panel explores a similar question, but in terms of number of tokens instead of LLM queries (we note that the blue and red lines are the same in shape, but the x-axis is now different and is measured in tokens, each generation on the red and blue lines is 512 token long, we now clarified that in caption). We clearly see that for a fixed budget of generated tokens having a larger number of shorter responses is most useful for effective separation. The best particular setup in Figure 4 is to use 10 task prompts with 50 generations, each 64-token-long, which is discussed in Section 5.2.

Thank you for your thorough review and constructive feedback. We appreciate your recognition of our novel application and the strengths you identified. We address your concerns below:

1. [Motivation for statistical framework and the use of permutation test.] Thank you for your question.

• First, we note that the problem of prompt membership inference (i.e., verification if a known system prompt is used by a model) is different from existing prompt reconstruction methods (for example, [1,2,3]) which aim to recover the prompt. While these methods achieve impressive results on prompt reconstruction given the difficulty of the reconstruction problem, their reconstruction success rate is not high enough to be able to confidently verify the prompt reuse and solve for prompt membership inference, they are computationally expensive usually relying on GCG-style optimization [4], and some of these methods require access to model gradients [3].

• Specifically, [1] achieves 44.6% to 74.1% success rate in prompt reconstruction (per Table 1 in [1]). [2] relies on knowing the target domain of system prompts and outperforms other reconstruction approaches in relative terms if the domain is known, but still achieves only 67.3% or 68.9% success rates in absolute terms on more difficult target models and domains (per Table 3 in [2]). Additionally, [2] reports that PLeak successfully reconstructs only 68% of prompts in real-world chatbots. [3] achieves up to 82% reconstruction success rate (per Figure 5 in [3]) but requires access to model gradients for optimizing adversarial prompts.

• For direct comparison, we ran additional experiments comparing PLeak [2] – the most high performing of the existing prompt reconstruction approaches (referenced in your and reviewer L9Uj’s comments) to Prompt Detective in the prompt membership setting. We observe that Prompt Detective is significantly more effective than PLeak in the prompt membership inference setting and report the detailed results of this experiment in our general response as well as in our response to your point 4 [Comparison to prompt extraction baselines] below. We added these results to Appendix B.1 as well.

• The above motivates the development of Prompt Detective as a statistical framework specifically tailored to the prompt membership inference setting. Prompted by your question, we expanded our original discussion on this in lines 58-63 of Introduction and section 2.2 of related work. Regarding the choice of the permutation test, we chose it for its non-parametric nature and ability to handle high-dimensional data without distributional assumptions. Compared to the latency of language model inference, the latency of running many permutations (usually thought of as the main downside of this test) was negligible.

2. [Condition in line 184 (Algorithm 1).] We appreciate your careful review of our algorithm. However, the original condition s* ≤ s_obs is correct for the following reasons: (1) Lower cosine similarity indicates more difference between the prompts, (2) We want to determine if the observed similarity is significantly lower than what we would expect by chance. (3) We count how often the permuted data gives a result as extreme or more extreme (i.e., as low or lower) than the observed data, (4) If we find few cases where s* ≤ s_obs, it results in a low p-value, indicating that the observed similarity was significantly low and extreme. This aligns with standard permutation test methodology where we count the number of permutations that produce a test statistic as extreme as or more extreme than the observed value.

3. [Null hypothesis.] We note that adopting the null hypothesis (H₀) which posits that the two samples originate from the same distribution is the standard setup in statistical hypothesis testing for determining whether two samples derive from different distributions. This framework is foundational to a wide array of established tests, including the Kolmogorov-Smirnov test, the Mann-Whitney U test, and the Anderson-Darling test, among others. However, we acknowledge that false positives (i.e., incorrectly predicting that a system prompt was stolen when it was not) are important and they carry more severe consequences. To mitigate this risk, we place significant emphasis on the power of our statistical test and conduct extensive experiments using challenging examples. In these scenarios, system prompts may differ by only a few words, yet Prompt Detective can still effectively distinguish between the distributions of the outputs generated by these prompts when a large number of generations (500) is utilized in the method.

Dear Reviewer DuyU,

Thank you for your valuable time and effort in reviewing our paper and rebuttal—we truly appreciate your thoughtful feedback.

Your insights have helped us improve our work, we have made a significant effort and added new experiments and results to address your concerns in the updated draft.

We kindly hope you will consider raising your score if our revisions and rebuttal have resolved your concerns. Please reach out with any further questions. Thanks again for your contribution to our paper.

Thanks to the authors for their rebuttal. It addresses several of my concerns and I mention the remaining ones below.

1. I think the motivation for statistical testing is still missing. I was not only curious to see the empirical advantage of the proposed method, but the reason for choosing statistical methods for prompt membership inference. Why not design a better empirical method?
2. I see a contradiction in this statement: "Lower cosine similarity indicates more difference between the prompts". Being a similarity metric, it should imply less difference. Either the authors should clarify this more in the paper or change their terminology. Currently, I don't understand their algorithm.
3. I think using the null hypothesis of the paper, just because general statistical methods adopt some conventions, is not justified. Being for a specialized application with particular consequences, the work should (re-)design every component of the framework according to the needs of the application.
4. I do not understand the reason given by the authors for universal task probes.

Thank you for your thoughtful review and positive feedback! We appreciate your recognition of the importance of our work and the clarity of our presentation. We are glad you found the paper well-written and easy to understand. Please, see our answers to your questions below:

1. [Adaptive attacks:] Thank you for this point. Indeed, the adversary rephrasing the system prompt is a possible scenario. However, we note that if the system prompt is rephrased such that the output distribution remains highly similar to the original output distribution (which would correspond to preserving utility), that will require a very large set of task prompts and responses to achieve low p-values and separation in Prompt Detective. Therefore, observing no separation until a very large sample of responses is used will be indicative of the adversary using such adaptive attacks attempting to preserve the output distribution (i.e. utility) while rephrasing the system prompt, so Prompt Detective will remain effective in this scenario. This is consistent with our hard setup experiments in Section 5.2.

2. [Embedding models:] We appreciate this suggestion. We have now conducted additional experiments using different embedding models (e.g. gte-Qwen2-1.5B-instruct, jina-embeddings-v3 and mxbai-embed-large-v1 models from MTEB leaderboard [1]) on the Awesome-ChatGPT-Prompts dataset and found that our results are consistent across various encoder models. We have included these new results in the Appendix B Table 4 and briefly mentioned them in the main text. Please, find the results below:

3. [Verification step in jailbreaking attacks:] While we consider Prompt Detective to be a defense mechanism enabling users to verify their system prompt reuse by adversaries, the method can indeed be used as a verification step on top of prompt extraction attacks. In fact, this research direction may lead to significantly improved prompt extraction attacks, however, within the scope of this paper, we acknowledge this fact as a potential ethical consideration for developing Prompt Detective in our Ethics Statement.

4. [Applicability to soft prompts and in-context demonstrations:] Prompt Detective is indeed directly applicable to the use cases of soft prompt and in-context example verification since it detects differences in output distributions induced by prompts. However, we decided to focus our exploration in this paper on system prompts. We note that using Prompt Detective as a membership inference attack to verify the use of a single in-context example or an unordered set of multiple in-context examples is a slightly more challenging task requiring additional care in test result interpretation as even changing the order of in-context examples may lead to subtle (or sometimes more pronounced) distribution shifts.

Thank you again for your positive review of our paper, and we hope our clarifications address your questions!

References:

[1] Muennighoff, N., Tazi, N., Magne, L. and Reimers, N., 2022. MTEB: Massive text embedding benchmark. arXiv preprint arXiv:2210.07316. (https://huggingface.co/spaces/mteb/leaderboard)

Thank you for your detailed review and constructive feedback. We appreciate your positive remarks regarding the relevance of our problem statement, the soundness and simplicity of Prompt Detective, and the overall clarity of our presentation. Below, we respond to each point of feedback in detail:

1. [Hypothesis test controls the wrong thing:] We note that adopting the null hypothesis (H₀) which posits that the two samples originate from the same distribution is the standard setup in statistical hypothesis testing for determining whether two samples derive from different distributions. This framework is adopted in a wide array of established tests, including the Kolmogorov-Smirnov test, the Mann-Whitney U test among others. However, we acknowledge that false positives (i.e., incorrectly predicting that a system prompt was stolen when it was not) carry more severe consequences. To mitigate this risk, we place significant emphasis on the power of our statistical test and conduct extensive experiments using challenging examples in Section 5.2. In these scenarios, system prompts may differ by only a few words, yet Prompt Detective can still effectively distinguish between the distributions of the outputs generated by these prompts when a large number of generations (500) is utilized in the method.

2. [The paper should also report full ROC curves:] Thank you for your suggestion, we have added the ROC curves for Prompt Detective in Appendix B Figure 6. We observe that Prompt Detective achieves ROC-AUC of 1.0 in all cases except for Claude on Awesome-ChatGPT-Prompts dataset where it achieves 0.98 ROC-AUC.

3. [Figures 3 and 4 lack positive pairs:] Thank you for your suggestion, we have now updated Figure 3 to include plots of p-values for positive pairs, however our experiments with hard examples do not include positive pairs, please see the explanation in the next comment. For Prompt Detective, the false negative rate (FNR) for positive pairs is determined by the chosen significance level. Our experiments confirm that the FNR remains close to 0.05 when using a significance level of 0.05. Similarly, adjusting the significance level to 0.01 results in an FNR close to 0.01, demonstrating consistency with theoretical expectations. In Figure 3 we demonstrate the relationship between the power of the test (False Positive Rate) and the number of queries used in the test, which does not affect the significance level and therefore the false negative rate. As expected, the average p-value in the positive case stays close to 0.5.

4. [Hard setup lacks positive pairs:] As noted in your review, the consequences of a wrongful accusation are significantly higher, and, to address this, the Hard Setup section focuses on evaluating the potential limitations of Prompt Detective when distinguishing between highly similar but distinct prompts. Including results for positive pairs would not offer additional insights, as this experiment has already been explored in the Standard Setup using the exact same prompts from the Anthropic Library. In the Hard Setup section, we specifically vary the "counterfactual prompt" while keeping the original (proprietary) prompt fixed, which are the same as ones in the Standard Setup.

5. [Why does the standard evaluation setup described in Sec. 4.3 only consider on negative pair instead of all possible pairs?:] Considering all possible negative pairs would result in approximately 30,000 prompt pairs. Running Prompt Detective on each pair would require 1,190,450 queries per model, which is computationally prohibitive. To address this, we randomly selected a subset of negative pairs equal in number to the positive pairs, corresponding to the total number of prompts in the dataset.

6. [Full results for typo case study in App. C.2:] Since these case studies include a single prompt, we can not report distribution level metrics such as FNR and FPR. However, we observe that Prompt Detective can distinguish between these prompts achieving p-value of 0.02 when a larger number of queries (1000) is used.

In response to the other reviewers' feedback, we conducted additional experiments, including (1) comparing Prompt Detective against PLeak, a prompt extraction baseline, and (2) an ablation study on Prompt Detective embeddings. Please, see our general response for details.

We sincerely thank you once again for your thoughtful feedback. We hope our responses have addressed your questions, and kindly ask you to consider increasing your score in light of the clarifications and improvements we have provided.