The Canary’s Echo: Auditing Privacy Risks of LLM-Generated Synthetic Text

We thank the reviewer for their feedback. We provide responses for the concerns raised below.

(1) one could imagine finetuning on D_tilde and then applying standard MIAs to the finetuned model

Many thanks for pointing this out. We have opted to train an n-gram model on the synthetic data rather than a larger, transformer-based model due to its simplicity and computational cost. Indeed, training the n-gram model on the synthetic data, both for SST-2 and AgNews takes less than 1 CPU minute. We will add the suggestion of training a full LLM on the synthetic data instead to the discussion section.

(2) include a discussion on the privacy risks of D

We provide results on fully in-distribution canaries (randomly sampled from D, no out-of-distribution suffix or F=max) throughout our work (in-distribution results in Table 1, and results for F=max in Figure 2 (c,f) and Table 2). Due to the lower perplexity of in-distribution sequences, we recover that data-based MIAs work quite well for these samples, especially compared to the high-perplexity canaries commonly used for model-based attacks. However, in these experiments, we consider the member canaries repeated n_rep times in the training data with n_rep up to 12 and 16. From Figure 2(a,d), we learn that when n_rep decreases, the MIA performance drops to no better than a random guess baseline. We hence anticipate that, at least in our experimental setup, the privacy risks associated with sequences appearing only once in D remains low. We will elaborate on this in the discussion section. We believe this also answers the reviewer’s last question (i.e. reporting the performance of the data-based MIAs on non-canaries belonging to D as a baseline).

(3) Could you report CIs for some of the major results

For our main results (e.g. Table 1), we report ROC AUC as the performance of the MIA, representing an average performance over all (1000) canaries.

Getting meaningful confidence intervals for these results requires training multiple (10+) target models, which is computationally quite expensive and was not feasible within the rebuttal period. We will run this for the SST-2 results in Table 1 to be included in a final version of the paper.

(4) The text in Figure 3a is quite small; could you make the plot more legible?

Thanks for pointing this out, we will increase the corresponding fontsize.

(5) Could you explore the effect of training data size on the success of the MIAs?

We share the reviewer’s intuition that as the size of the training dataset increases, the MIA performance likely decreases. As part of the rebuttal process, however, we have prioritized running other experiments and would leave this analysis to future work.

We thank the reviewer for their feedback. We provide responses for the concerns raised below.

(1) A minor limitation is that the experiments are conducted on only two datasets

We provide results for the n-gram based MIA for the SNLI dataset (for the setup from Table 1) below and will include this in the paper. These results suggest the same trends we report carry over to other datasets.

(2) How do attacks work on differentially private text generation?

We will add a section on mitigations strategies for our novel MIAs, focusing on fine-tuning the target model with DP-SGD before generating synthetic data, as in prior work (Yue et al., 2023; Mattern et al., 2022; Kurakin et al., 2023). Given past results (Table 3 in [1], Figure 3 in [2], or the results of the SaTML 2023 Membership Inference competition on STT-2 [3]), we expect the performance of model-based attacks to quickly decrease to a random guess baseline under DP guarantees. Since data-based attacks underperform compared to model-based attacks, and guarantees transfer to the synthetic data due to DP’s post-processing property, we expect data-based MIAs to also approach random guess for practical values of ε. By the end of the rebuttal phase, we aim to provide meaningful ablations on MIAs against DP-synthetic data, which we will then include in the paper.

We leave other defense strategies (e.g. using paraphrasing techniques) as proposed by the reviewer for future work, and will elaborate on this in the discussion section.

[1] Xie, et al. Differentially Private Synthetic Data via Foundation Model APIs 2: Text

[2] Ma et al. Efficient and Private: Memorisation under differentially private parameter-efficient fine-tuning in language models.

[3] Microsoft Membership Inference Competition (https://github.com/microsoft/MICO).

We thank the reviewer for their feedback. We provide responses for the concerns raised below.

(1) analyzing how different domains might impact the auditing efficiency

In Table 1, we also study the effect of which labels (or domains) the canaries belong to. In particular, we consider both ‘natural’ and ‘artificial’ labels associated with canary samples, where ‘natural’ corresponds to labels from the same distribution as the labels from the original dataset and ‘artificial’ corresponds to a new, canary-specific label (see Sec. 4). We observe a slight increase in MIA performance across all data-based MIAs, suggesting that more rare, potentially artificially crafted labels make canaries more vulnerable. We leave a more thorough study on this effect to future work and will add this to the discussion section.

(2) measuring the leakage against privacy-preserving methods

We will add a section on mitigations strategies for our novel MIAs, focusing on fine-tuning the target model with DP-SGD before generating synthetic data, as in prior work (Yue et al., 2023; Mattern et al., 2022; Kurakin et al., 2023). Given past results (Table 3 in [1], Figure 3 in [2], or the results of the SaTML 2023 Membership Inference competition on STT-2 [3]), we expect the performance of model-based attacks to quickly decrease to a random guess baseline under DP guarantees. Since data-based attacks underperform compared to model-based attacks, and guarantees transfer to the synthetic data due to DP’s post-processing property, we expect data-based MIAs to also approach random guess for practical values of ε. By the end of the rebuttal phase, we aim to provide meaningful ablations on MIAs against DP-synthetic data, which we will then include in the paper.

[1] Xie, et al. Differentially Private Synthetic Data via Foundation Model APIs 2: Text

[2] Ma et al. Efficient and Private: Memorisation under differentially private parameter-efficient fine-tuning in language models.

[3] Microsoft Membership Inference Competition (https://github.com/microsoft/MICO).

(3) motivation for using n-gram for data-based attacks is not clearly described

Many thanks for pointing this out. We have opted to train an n-gram model rather than a small, transformer-based model due to its simplicity and computational cost. Indeed, training the n-gram model on the synthetic data, both for SST-2 and AgNews takes less than 1 CPU minute. We further provide ablations of the value of n in Appendix H and Table 10, where we consistently find n=2 to be the optimal value. We will add the suggestion of training a small neural network instead to the discussion section.

(4) How does the size of synthetic data impact the auditing?

We provide ablations for different sizes of the synthetic data in Appendix H and Figure 5. While we observe an improved performance for the n-gram based MIA as more synthetic data is generated, we maintain our main analysis considering a synthetic dataset of equal size to the private dataset as this is more realistic and used in prior work (Yue et al., 2023; Mattern et al., 2022; Kurakin et al., 2023).

(5) How do different embedding models impact the auditing?

For similarly-based methods, we opted for paraphrase-MiniLM-L6-v2 from sentence-transformers as the embedding model, as it offers great performance in semantic search benchmarks link. As the performance of the MIA based on semantic similarity using this embedding model is outperformed by all other data-based MIAs (Table 1), we did not further ablate the choice of the embedding model. We leave this to future work and will add this to the discussion section.

Thanks for the feedback; we provide detailed responses below.

Difference between model-based and data-based MIAs supported only through experiments on two datasets. A large set of variations of these experiments have been conducted, which demonstrate the robustness of the proposed approach.

We are glad the reviewer thinks our experiments demonstrate the robustness of our approach. We chose to investigate the gap between model- and data-based attacks in depth through detailed ablation studies rather than shallowly on a broader range of datasets.

We provide results for the n-gram based MIA on SNLI (same setup as Table 1), which we will include in a revision. Results suggest the same trends we report carry over to other datasets.

Difference between model-based and data-based attacks not well explained

We thoroughly describe the difference between model- and data-based MIAs in Sec. 2, including pseudocode in Alg. 1. Appendix E contains additional discussion on the difference between the threat models. We present concrete model- and data-based attacks in Sec. 3.1, including the calculation of membership inference signal and our adaptation of RMIA. We detail the choice of hyperparameters and how we evaluate attacks in practice in Sec. 4. We include additional details about model- and data-based attacks in Appendices A, B.

We welcome suggestions to make this clearer if what we already provide is not enough.

The process for generating a synthetic text is unconventional

We generate synthetic data as done conventionally (Yue et al., 2023; Mattern et al., 2022; Kurakin et al., 2023), by prompting the model on label-dependent templates (cf. Appendix C), so that text is generated based on the indication of the prompt.

The fact that the synthetic dataset should be the same size as the original is not realistic

We report on experiments on synthetic datasets 2-8x larger than the original in Appendix I.

We focus on generating synthetic data derived from a private dataset so that e.g. downstream tasks on the synthetic dataset have similar utility. Hence, in our main experiments we generate datasets matching the size and label histogram of the private data. This is the same setting studied by Yue et al., 2023; Mattern et al., 2022; Kurakin et al., 2023.

Possibility that the training set and the canaries have been used during pre-training

Mistral-7B's training data is not public, so we cannot rule out SST-2/AG News being included in the training data. However, we aim to measure the difference in performance between model- and data-based MIAs on the fine-tuning dataset of a model used to synthesize data, which training and fine-tuning data overlap would affect similarly.

We can, however, rule out the presence of canaries in the training data when they are constructed artificially (in-distribution prefix F=0; Table 1, Fig. 1(a,b,d,e), Fig. 2). Especially at high perplexities, these canaries are likely absent from the training data. For canaries with F>0 and if parts of the datasets were included in pretraining, it would, if anything, make MIAs more challenging. We will include this discussion in an eventual revision.

No justification of the choice of parameters used for the two variants of the MIA proposed.

We discuss hyperparameter selection in Appendix H (paragraph “Hyperparameters in data-based attacks”). We consistently find the best performance for n=2 (n-gram MIA) and k=25 (number of closest synthetic records for similarly based MIAs), which we used in the main experiments.

differentially-private variants of the model training.

We will add a section on mitigations, focusing on fine-tuning the target model with DP-SGD before generating synthetic data as in prior work (Yue et al., 2023; Mattern et al., 2022; Kurakin et al., 2023). Given past results (Table 3 in [1], Figure 3 in [2], or the results of the SaTML 2023 Membership Inference competition on SST-2 [3]), we expect performance of model-based attacks to decrease to a random guess baseline under DP. Since data-based attacks underperform compared to model-based attacks and guarantees transfer to synthetic data due to DP’s post-processing property, we expect data-based MIAs to also approach random guess for practical values of ε. By the end of the rebuttal phase, we aim to provide meaningful ablations on MIAs against DP-synthetic data, which we will then include in the paper.

[1] Xie, et al. Differentially Private Synthetic Data via Foundation Model APIs 2: Text

[2] Ma et al. Efficient and Private: Memorisation under differentially private parameter-efficient fine-tuning in language models.

[3] Microsoft Membership Inference Competition (https://github.com/microsoft/MICO).

Fig.5 legibility.

Thanks for pointing this out, we will address this.