Privacy-Preserving In-Context Learning with Differentially Private Few-Shot Generation

We thank the reviewer for their careful reading and thoughtful comments along with their positive score.

In response to the reviewer's comment about the resource consumption involved in generating a composite prompt by consulting the large model M*L times, we acknowledge and agree with this observation. However, it is crucial to highlight that this process represents a one-time cost that is incurred prior to the deployment of the model. Once this initial stage is completed, it enables the model to be used indefinitely without incurring additional costs in this regard.

We recognize the concern raised about the reliance on an untrusted model to generate synthetic few-shot demonstrations. It may be the case that open-source models might not always match the generation capabilities of their closed-source counterparts. Nonetheless, it's important to highlight the rapid advancements being made in the capabilities of open-source models. Recent progress (with open-source LLMs such as LLama-2, Falcon, Vicuna, Mistral, among others) indicates substantial improvements in these models, suggesting a promising future where they could potentially rival the performance of their closed-source counterparts. We also want to emphasize that investigating the use of open-sourced models represents a crucial and intriguing avenue for future research. Delving into this area could effectively address the issue of reliance on an untrusted model.

We thank the reviewer for their question regarding the quality of the synthesized prompt. In this work, our primary focus is on in-context learning, and we believe that the performance of models with synthetic few-shot demonstrations during this process is a strong indicator of the prompt's quality. However, we concur with the reviewer that aspects such as length, coherence, and fluency are critical in determining the overall quality of the synthesized prompt. While a comprehensive evaluation of these aspects slightly exceeds the scope of our current study, which is centered on in-context learning, we acknowledge the importance of these factors. In response to the reviewer's question, we additionally include sample few-shot generations by Babbage from randomly selected experiments in Appendix H of revised manuscript. These samples provide additional insights into the quality of the synthesized prompts, particularly in relation to the aspects highlighted by the reviewer. We note that while the Babbage model is comparatively smaller and less advanced than the current state-of-the-art models, its synthetic generations maintain a reasonable level of coherence and fluency. Generally, the length of these synthetic generations aligns with the samples in their respective training datasets. However, we have noticed that longer generations occasionally result in the repetition of the last sentence. We believe that this issue can be mitigated by employing more capable instruction-following models and refining the instruction during the generation process.

We thank the reviewer for their careful reading and thoughtful comments along with their positive score. We address all their questions below.

1. As the reviewer correctly pointed out, indeed Algorithm 1 incurs M foundation model API calls per token generation. Along with M, the total cost of a synthetic sample generation also depends on the number of tokens in the prompts and the number of generated tokens. We present the monetary costs associated with applying Algorithm 1 for the results in Table 3 to provide readers with a clear understanding of the expenses incurred for different models, including Ada, Babbage, Curie, and Davinci. We highlight that these monetary costs represent one-time expenses incurred prior to the deployment of the model. We add the discussion on monetary cost in Appendix G and would be happy to engage in further discussions with the reviewer should any follow-up suggestions arise.

2. In our paper, we investigate two mechanisms: (i) Gaussian DP mechanism and (ii) Report-Noisy-Max with Exponential DP mechanism. We compare both mechanisms in Table 4 and observe in general competitive performances between the two DP mechanisms. For both mechanisms we use the argmax sampling (i.e., report-noisy-max) to generate tokens. Therefore, we can state that they perform similarly in our experiments.

However there is an important difference between the two mechanisms. Report-Noisy-Max with Exponential DP mechanism can only work with argmax sampling to generate tokens, however, line 14 of Algorithm 1 can be utilized to improve the maximum probability to be 1 thanks to the fact that only the argmax token is reported. Unfortunately, this benefit cannot be instantiated with the Gaussian DP mechanism. In other words, Report-Noisy-Max with Exponential DP works as long as sensitivity is bounded in \ell_\infinity norm. But Gaussian Mechanism (even if only reporting argmax token, i.e, RNM with Gaussian Noise) requires the sensitivity to be bounded in \ell_2 norm, which is a stricter requirement. However, the advantage of the Gaussian DP mechanism is that it benefits from the post-processing of DP, therefore, more advanced sampling techniques for token generation can be used instead of the basic argmax sampling. We leave the exploration of this to future work.

We thank the reviewer for their careful reading and thoughtful comments along with their positive score. We offer the following detailed discussion in response to the reviewer's comments.

We agree with the reviewer that researching the comparative effectiveness of DP few-shot generation versus DP fine-tuning is a compelling and important area of study. The appropriateness of either approach may hinge on a variety of factors, including the nature of the downstream task and the level of access to the model. For instance, direct fine-tuning with DP might not be feasible in scenarios where one is limited to API-level interactions with the model. Conversely, certain downstream tasks might not yield optimal results with mere few-shot demonstrations, potentially benefiting more from fine-tuning for enhanced adaptation to specific domains. In light of these considerations, we recognize the importance of future research aimed at elucidating the circumstances under which DP few-shot generation or DP fine-tuning would be more advantageous.

We thank the reviewer for the insightful comment regarding the potential of a stronger data-dependent privacy analysis similar to the prior work on PATE that utilizes the level of agreement among the teachers. Indeed this improvement would be beneficial for Algorithm 1 because to obtain strong privacy levels (i.e. low epsilon values), we apply resampling the demonstrations from the private dataset for each token generation that benefits the amplification of privacy by subsampling theorem. Therefore, stronger data-dependent privacy analysis may benefit overcoming the need for resampling per token generation and in general provide us with improved privacy levels. We agree that this presents an intriguing direction for future research.

We thank the reviewer for their careful reading and thoughtful comments. We address all their questions below.

1. Thank you for bringing to our attention the important aspect of testing our results against real-world threat models. We acknowledge the reviewer's perspective regarding empirical evaluations that test epsilon-utility trade-offs against actual attacks as such assessments lead to a more holistic understanding of privacy and utility.

To address this, we follow the prior work (Duan et al. 2023) where the authors instantiate membership inference attack (MIA) in the framework of in-context learning (ICL). The goal of their attack is to determine if a given data point was used within the prompt of the LLM. Perhaps expectedly, using actual samples from the private dataset leads to successful MIA results. To apply MIA for our DP solution, we take the following approach. We study the DBPedia dataset and split the dataset in two parts for member and non-member samples. We use the member samples to generate synthetic demonstrations with our DP algorithm. Similar to Duan et al, we consider 1-shot ICL on the Babbage model and generate 1-shot demonstrations in 5 independent runs for $\epsilon=[1, 2 ,4 ,8]$. During ICL, we attempt MIA for the samples from members and non-members and similarly calculate the TPR/FPR. For each DP generated demonstrations, we run 20 trials for MIA and we finally average across the 100 trials to calculate the AUC for each epsilon. For the MIA against non-private 1-shot ICL, we follow Duan et al.(2023) and average across 100 trials for AUC.

We present the membership privacy attack result with the AUC metric in the table below.

Similar to Duan et al. (2023), we observe that using actual samples from the private dataset leads to successful MIA results (81.84 corresponding to $\epsilon=\infty$). On the other hand, our DP solution reduces the MIA AUC to almost random guesses, demonstrating the effectiveness of our privacy-preserving methodology for ICL. It is interesting to observe that even for a theoretically large epsilon value such as 8, MIA attack remains close to random guess, which presents promising (empirical) privacy-utility tradeoffs. We add the detailed empirical privacy evaluation by MIA in Appendix F in the revised draft.

2. We thank the reviewer for pointing out this statement in our paper. We acknowledge the importance of accurate discussion and appreciate the reference to the recent study the reviewer provided. We propose the following modification (along with adding more related work) and we would be happy to revise this in further discussions with the reviewer should any follow-up suggestions arise:

“We finally review an active area of research on the generation of sanitized text documents. In this domain, Feyisetan et al. (2020); Xu et al. (2020); Carvalho et al. (2021); Du et al. (2023) add noise at the word level and use metric local differential privacy (Chatzikokolakis et al., 2013). Mattern et al. (2022b); Utpala et al. (2023) directly operate on documents using local differential privacy (LDP) (Kasiviswanathan et al., 2010; Duchi et al., 2013) where the former is based on fine-tuning for paraphrasing and the latter uses zero-shot prompting for paraphrasing followed by sanitization.”

We would very much appreciate the reviewer considering increasing their rating in case they find our responses compelling.