PANORAMIA: Privacy Auditing of Machine Learning Models without Retraining

Thank you for the thoughtful review.

How is the helper model used in the baseline trained? (Q1)

The helper model has the same classification task and architecture as the target model. To train it, we generate separate sets of training and validation data using our generator. For image data, the synthetic samples do not have a label. We thus train a labeler model, also a classifier with the same task, on the same dataset used to train the generator. We use the labeler model to provide labels for our synthetic samples (training and validation sets above). We train the helper model on the resulting training set, and select hyperparameters on the validation set. We give more experimental details for each modality in App. C (l. 473-474, 545-546, 567-568, 586-592). App. D1 further highlights the various architectures of the helper model we tried (l. 638, Table 5), including training the helper model on real non-member data. Table 5 shows the helper model trained on generated data gives the strongest baseline (highest $c_{lb}$).

The lexicographic order in Eq. (2) appears to be essential for Corollary 2 and seems to be aimed at ensuring something like $\epsilon > c$ to hold [...]? (Q2)

The lexicographic order in Eq. (2) is not to ensure $\tilde{\epsilon} > c$ (as a counter-example, $(c_1=3, \epsilon_1=1) \leq (c_2=3, \epsilon_2=2)$). The technical step relying on this order is the construction of the confidence interval (Corollary 2, l. 187-188) based on hypothesis tests from Prop. 1 and 2.

The reason for this order with $c$ first is that Prop. 2 asks: ``if the generator is $c$-close, is the target model $\epsilon$-DP?'' We thus must reject any hypothesis with a $c$ we can disprove based on data, before computing $\epsilon$ based on a plausible $c$. Without this order (e.g. $\epsilon$ first or $c + \epsilon$), we would not necessarily compute $\epsilon$ based on a plausible $c$. We would assign some of the MIA performance to privacy leakage, when we know for a fact it comes from the generator being too far from the data distribution.

How exactly does the formulation in the paper differ from the setting introduced by Steinke et al., 2023 [...]? (Q3)

The key difference lies in how we create the audit set. In Steinke et al., 2023, the audit set is fixed, and data points are randomly assigned to member or non-member by a Bernoulli random variable $S$. Members are actually used in training the target model $f$, while non-members are not (so assignment happens before training). In our framework, we take a set of known iid. members (after the fact), and pair each point with a non-member (generated iid. from the generator distribution). We then flip $S$ to sample which one will be shown to the ``auditor'' (MIA/baseline) for testing, thereby creating the test task of our privacy measurement.

Regarding the independence assumptions, $S$ is independent of everything by construction, as in Steinke et al., 2023. In fact, if we replace our generated data with in-distribution independent non-members, we exactly enforce the same independence, except we waste some data by drawing our auditing game after the fact. Using generated data adds another complexity, as $S$ ``leaks'' through the actual data-point shown to the auditor (based on differences between the generator and member data distributions), which we address with the baseline model.

Correctness of Proposition 2. (W1)

We hope Q3 above already helps resolve the misunderstanding about equation line 434 and Prop. 2. To expand: while $S$ is independent of $f$ ($S \perp f$), they are not conditionally independent when conditioning on $X$ (where $X$ is the test set for the MIA/baseline), so $f \not\perp S | X$. This is because $X_i$ is either a member or non-member based on $S_i$, and the member is a data point on which $f$ was trained!

Hence, Eq. below l. 434 does not evaluate to $1$. Deliberately ignoring $S_{<i}, X_{<i}$ for simplicity, the Eq. reads as the ratio between: (numerator) the membership guess (a post-processing of $f$ and $X_i$) conditioned on the event that we guess for input $X_i$ which is a generated non-member $S_i=0$; and (denominator) the membership guess (still a post-processing of $f$ and $X_i$) conditioned on the event that we still guess on $X_i$ which is now a member of $f$'s training set. If $f$ is DP, the membership guess is a post-processing of a DP result on two neighboring datasets ($X_i$ wasn't in the training set vs. $X_i$ was in the training set), and hence obeys the inequality shown after l. 434. Note that collisions (where we have $X_i, S_i=0$ but $X_i$ is also in the training data) make the ratio $1$: the inequality is still true and the theory applies, but the privacy measurement has no power (see discussion on overfit generator in Q1 of reviewer 55Rd).

Adjusting significance level does not give larger DP estimate values. (W2)

We adjust the significance level because we need to compare the highest lower-bounds implied by the MIA and the baseline. Notice on Fig. 3a and 4a that the highest precision values and implied lower-bounds can happen at different (unknown in advance) recall levels. If we were to fix a recall value in advance, we could get misleading results based on whether that value is closer to the best value for the baseline or for the MIA. We thus have to make the comparisons at different recall levels.

In this case our tests from Prop. 1, 2 (which are at a fixed number of guesses) need a union-bound $\beta \leftarrow \beta / m$ to be correct. We could select the maximum without a union bound as a heuristic, but in practice it barely changes the results. This is because both the baseline and MIA numbers are lowered in a similar way by their respective union-bounds, so the gap between the two remains very similar.

Auditing for $(\epsilon, \delta)$-DP algorithms. (W5)

We do analyze the $(\epsilon, \delta)$-DP case in App. E2, and show results in Table 10 in App. E3.

Thank you for the thoughtful review. Hereafter, we start with answers to the questions, before addressing other weaknesses listed that we believe stem from a miscommunication.

Questions:

Q1: Might there be a way to use the membership classifier itself as a baseline, e.g., by somehow averaging over possible values for model statistics? This would avoid skewed scenarios where the MIA is better at detecting synthetic data than the baseline detector (even when ignoring membership signal).

Thanks for this interesting suggestion. If we could somehow capture a distribution of "non-member losses" for this data point, we might be able to marginalize out the dependency of the MIA on the target model and extract the input-dependent part. A key challenge with such a design would be to make sure that the resulting baseline is indeed as strong as it can be as well as that input differences do not ``leak through'' the target model loss for member data. In our experiments, the baseline is often better at picking up signal from the data point than the MIA model is (hence the cases, like in Figure 5 or some of the tabular data plots in the appendix, in which the baseline outperforms the MIA), which is in a sense the opposite problem. We believe it to be an interesting future work, and in general, any progress on discriminative models and MIAs will directly plug-into and benefit our approach.

Q2: In Figure 6/Section 5.3, do all models use (approximately) the same number of training samples? If not, could there be confounding effects (e.g., if models trained on fewer samples leak more privacy)?

Experiments for all data modalities in S5.3 show the effect of varying test set sizes on the performance of both Panoramia and O(1) while keeping all other experimental variables, including training dataset size constant across all the models. This is to ensure the results we observe are primarily due to varying the test dataset size while keeping everything else controlled. We mention the training details for the models in Appendix C. In addition, we also show, in Appendix D4, how varying training data for a fixed test set size impacts both Panoramia and baseline performance.

Q3: Is there a path to extend this paper's method to always yield a lower bound on (w.h.p.)? Or would this require radical changes?

Great question! This is plausible, and we hope that we, or another group, will figure out how in the future. The most straight-forward way to get a proper lower-bound on $\epsilon$ in our framework is to figure out how to measure (or construct) an upper-bound on $c$, as then $(c+\epsilon) _{lb} - c _{\text{ub}}$ yields $\epsilon _{lb}$. Getting an upper-bound on $c$ in the general case seems challenging though. There might be a way to leverage DP to do this by construction (since DP bounds the gap between distributions), though we haven't figured out how to do it yet. This is definitely a promising avenue for future work.

Other important clarifications:

Weakness 2 - Relatedly, the definition of c-closeness (Definition 3) might not require a generator to capture the tail of the data distribution. However, this tail contains outliers that are often particularly vulnerable to privacy leakage. Hence, ignoring such outliers might significantly underestimate the lower bound. I would have appreciated a more thorough discussion of this limitation (or a clarification).

In cases where the generative model is unable to effectively capture the long tail distribution of the real dataset it is modeling, it becomes easier for a baseline classifier to distinguish between real and synthetic non-members, and hence to tell the two distributions apart ($c$-closeness, like pure-DP, does not allow such discrepancies in the tail, though our $(c, \gamma)$-closeness relaxation in appendix E does). In practice, we do observe this phenomenon, especially with tabular data. We highlight this limitation via results on the tabular setting in Appendix D6 (we will add a pointer to the paper for clarity). In such a case, we show that the baseline is as strong or stronger than the MIA and Panoramia is unable to effectively detect any privacy leakage from the respective target model.

Weakness 3 - The paper could be more explicit about whether PANORAMIA is intended to be a procedure to be used in practice, or an important stepping stone towards more practical procedures. Right now, PANORAMIA achieves worse results than the existing (1) method in a white-box setting, and requires training of a strong synthetic data generator (which might be non-trivial). Nevertheless, I believe this paper is still relevant from a conceptual perspective as a path for future work.

Thank you for the helpful feedback. We will clarify that we believe that our work is an important step towards solving the privacy measurement setup that we tackle, namely measurements without control of the training process and with distribution shifts between members and non-members. The end-goal for future work is a full-fledged approach (that provides a proper lower bound) with potential improvements over other approaches (e.g., optimizing the generator to improve the audit). However, we also believe that our framework can already be useful, for instance for providing improved measurements with more data (Fig. 6a); or to tackle privacy measurements in models for which there are no known in-distribution non-members, which as recently pointed out in [1] suffers from the same distribution shifts we tackle by using out-of-distribution non-members (our theory would apply to such a case, though whether the generative model part can help is still an open question).

[1] Das, Debeshee, et al. "Blind Baselines Beat Membership Inference Attacks for Foundation Models." 2024. https://arxiv.org/abs/2406.16201

Thank you for the thoughtful review.

Q1: What happens when you develop a generator that is perfect (c=0) which just samples randomly from the known member subset.

In that case (a highly overfitted generator), the baseline and MIA will output $c _{lb} = \\{\epsilon+c\\} _{lb} = 0$ (though technically $c \neq 0$, since $c$-closeness quantifies the distance between the generator's distribution, and the distribution of members, not the specific member data points). This happens mainly because both the MIA and baseline detect leakage based on the same data, and assess the data in two ways: (1) the distributional difference in the data points themselves; and (2) for the MIA the difference in distributions of loss values the target model attributes to the member vs non-member data. In this case, our theory still applies but returns $\tilde{\epsilon} = 0$. Thus, we would not faultily ascribe privacy leakage to the target model, but the measurement would also fail to detect any real privacy leakage (one could see that the measurement is failing though, since $c _{lb} = \\{\epsilon+c\\} _{lb} = 0$).

Q2: More generally, Panoramia largely depends on a good generator and a baseline MIA. Can authors elaborate on the associated limitations? [C]an good generators be developed across all usecases (smaller datasets, data modalities etc)? And how should a baseline performance be developed or evaluated to be used as part of the panoramia framework?

The ability of the generator to capture well the data distribution is a key dependency of PANORAMIA and the reason why we have thoroughly evaluate our approach. In particular, in the tabular case, datasets typically contain a large number of ``average'' samples and a long tail of extreme values or rare classes. Thus in such a case, when the generated data does not capture the long tail, the baseline easily classifies all outliers as real data as no synthetic data points are similar to them. This leads to a strong baseline that is hard to beat for the MIA, which means that the privacy measurement then fails (though we can diagnose why). This is a limitation of our work that we discuss in details in App. D6 as a possible cause for the failure of the tabular data modality. In general, the baseline should be made as strong as possible to detect such failures of the generative model following the traditional ML best practices. For this reason, we have dedicated a lot of effort in designing and evaluating our baselines, including the helper models (S5.1 and App. D1).

Our design and theory offer practical advantages as well:

• As generative models improve, especially for smaller dataset sizes, so will our approach.
• The generator opens an interesting design space, in which one could try to optimize synthetic data for audit quality. While we have not yet explored this design space, we leave it as an interesting avenue for future work.
• Finally, our theory applies to other out-of-distribution non-member data, such as when using other datasets as non-members [1].

Q3: [T]he method authors provide does not give a formal lower bound for the privacy loss, but rather a proxy for it. [...] Why would [one] opt to compute an estimate for epsilon using your method? Instead, [one could] generate non-members in the same way as panoramia and just quantify the MIA performance with an AUC or TPR at low FPR compared to a baseline. [...] Authors should further motivate the relevance of a proxy for a lower-bound privacy loss in practice.

As it is the consensus in the field, we believe that DP provides the best semantics to define and quantify this type of privacy leakage. Thus, usually, privacy audits aim to provide a lower bound for the privacy loss. While we do not yet provide a lower bound, using DP semantics is still useful. For instance, [2] makes a convincing case that accuracy or AUC are not good metrics for privacy leakage. Rather TPR at low FPR is better, but notice that we need to compare the MIA with the baseline, and their performance most revealing of privacy leakage happens at different FPR values (cf. Figures 3a and 4a, though in precision/recall terms). In this case, we cannot directly subtract TPR values. Our theory tells us: (1) which FPR we should chose and (2) how to scale the TPR at this FPR (by mapping it to a $c$ or $c + \epsilon$ value) to make the values comparable between the baseline and the MIA.

Taking a step back, we hope that our framework will be a stepping stone for further research, maybe enabling a proper lower-bound on $\epsilon$ by measuring (or enforcing by construction) an upper-bound on $c$ (then $\\{c+\epsilon\\} _{lb} - c _{ub}$ yields $\epsilon _{lb}$). Though we do not know how to do it yet, this is a promising future work.

Q4: To further motivate post-hoc privacy auditing (without any control of the training data), I wonder if it makes sense to also emphasize the context of generative AI models such as LLMs? These models are increasingly trained on all the data model developers can acquire so the absence of non-member data is very real in practice.

Thanks for this great point. Actually, a paper made public after our submission [1] highlights this exact issue. As you mention, in this case, there is no known non-member data from the same distribution, MIA benchmarks use out-of-distribution non-member data (e.g., more recent datasets). Thus using generated data might yield better non-member data. Our theory applies equally to both real and synthetic non-members data and we believe that it may be an interesting building block for that setting.

Q5: Can authors add a related work section?

We propose to add a more formal related work section to collect the closest works, and an extended one in appendix.

[1] Das et al. "Blind Baselines Beat Membership Inference Attacks for Foundation Models." 2024. https://arxiv.org/abs/2406.16201 [2] Carlini et al. "Membership inference attacks from first principles." S&P 2022.

Thank you for the thoughtful review. We start with answers to the two explicit questions, before addressing two of the weaknesses listed that we believe stem from a miscommunication or misunderstanding.

Questions:

Q1 - Figure 3a suggests an ordering of 100 > 20 > 50 in terms of leakage for high-recall region, whereas numbers in Table 1 suggest that the trends should be like 100 > 50 > 20 if the proposed method is indeed measuring what it is supposed to. Why is that so?

In our experiments on all data modalities and models, the precision values that lead to the highest lower-bounds $c$ or $c + \epsilon$ are achieved at fairly low recall but for different recall values (especially for images). For this reason, when training the baseline and MIA models, we use a validation set to select hyper-parameters and training stopping time that maximize the lower bounds (effectively maximizing the precision in the lower range of recall values) on the validation set. We will clarify this in Appendix C. This is why for some instances like pictures, MIA models have the wrong order at high recall values. While they could probably be tuned to achieve higher precision and be ordered correctly, we do not do it since that regime is not where the highest lower-bounds happen. Indeed, we can see in both Figure 3 and associated results in Table 1 that in the recall regions at which the lower-bounds are maximal, the order is as expected (100 > 50 > 20).

Q2 - L480-481: this means the "labeler" is much weaker and might be generating incorrect labels more often? Are there any numbers for what the actual test performance of this labeler is?

The labeler used to train the helper model can indeed be weaker and generate wrong labels. For instance, our labeler for image data has 80.4% test accuracy on the CIFAR10 real data test set. The rationale for this approach is to augment the baseline with a model providing good features (here for generated data) to balance the good features provided to the MIA by $f$ outside of the membership information. In practice, the labeled generated data seems enough to provide such good features, despite the fact that the labeler is not extremely accurate. We studied alternative designs (e.g., a model trained on non-member data, no helper model) in Appendix D.1, Table 5, and the helper model trained on the synthetic data task performs best (while not requiring non-member data, which is a key point).

Other important clarifications:

Weakness 1 - [W]hy not use the non-member set as used in standard MIA setup (validation/test data). What is the added benefit from this extra step of generating synthetic non-members?

We do study this setup in Fig. 6, comparing a MIA using up to the full test data and our approach using generated data. On CIFAR10 we can observe that, for the same non-member dataset size, using real data performs better (when using an ML-based MIA instead of a loss-based attack in this case). However, generated data enables us to use more data points for training and evaluating the MIA (and baseline), leading to larger measurements of privacy leakage. This is true for both regular and DP models (Fig. 7a and 7c), despite the fact that the CIFAR10 test set is fairly large compared to the training set (20%) and such a large portion of training data may not be kept for testing in general.

Going in the same direction, a very recent work made public after our submission [1] shows that a similar member/non-member distribution shift issue happens when measuring privacy leakage from foundation models. In that case, the models are trained on vast amounts of data, and there is no known non-member data from the same distribution. As a result, MIA benchmarks use out-of-distribution non-member data (e.g., more recent datasets). Thus, using generated data might yield better non-member data. Additionally, our theory applies to both real and synthetic non-members data. Consequently, we believe that our approach may be an interesting building block for the setting described in [1].

Finally, we believe that the generator opens an interesting design space, in which one could try to optimize generated data for audit quality. We have not yet explored this avenue of research, but we believe that it is an interesting future work.

Weakness 3 - I have some concerns over the dependency on how well the baseline in/out distribution detection system works. As a concrete example consider Figure 7(a, b) - even as a human I see a very clear difference in resolution of the generated images and find it hard to believe that the distinguisher does not work well here. Even a non-ML technique (that can work around with blurring) would work pretty well here.

This is a great point, thank you. We apologize as this is a miscommunication on our end: our CelebA experiments are using a $128 \times 128$ resolution, thus our generative model generates images at that resolution. In Figure 7a though, we wrongly showed full resolution $218$ x $178$ images, hence the resolution difference. We fixed this figure to display the real images as we use them, and included it in the associated PDF answer (Fig. 1), for which we can see that the difference in resolution disappeared. Also note that the generator only needs to generate some good images, enough that it is hard to detect real members with high precision. This is because we play a one-sided member-detection game, as explained in Remark 3 ll. 137-144. Of course, it is always possible that much stronger baselines exist, although we did spend quite a bit of effort and time on making them as good as we could (see details in Appendix D of the submission).

[1] Das, Debeshee, et al. "Blind Baselines Beat Membership Inference Attacks for Foundation Models." 2024. https://arxiv.org/abs/2406.16201