Low-Cost High-Power Membership Inference by Boosting Relativity

Many thanks for the detailed comments.

1. Theoretical justification

We have revised Section 2. We substantially improved the organization of the section and provided interpretable presentation of our framework. First we present a template for the game and the MIA test which also applies to all major MIA methods in the literature.

MIA is a hypothesis testing problem that assigns a membership score $\mathrm{Score}_\mathrm{MIA}(x; \theta)$ to every pair of $(x, \theta)$, and outputs a membership bit through comparing the score with a threshold $\beta$:

$\mathrm{MIA}(x; \theta) = 1 \text{ if } \mathrm{Score}_\mathrm{MIA}(x; \theta) \ge \beta$

For any given threshold $\beta$, the adversary's power, defined as the true positive rate of the attack, and error or the false positive rate, are quantified over numerous repetitions of this experiment.

The ROC curves are computed by sweeping over all values of $\beta$.

Our work

We propose a novel method to compute $\mathrm{Score}_\mathrm{MIA}(x; \theta)$. The rest of the framework for the game and the test and their evaluation are the same as the prior work.

Revised Section 2

We have not made any change in the LR test, however, we made small changes to the way we arrive at the test which enables us to better explain our intuitions. The probability terms are also better explained. Here is the summary of the process:

We compute the pairwise likelihood ratio as $\mathrm{LR}\_{\theta}(x, z) = \frac{\Pr(\theta | x)}{\Pr(\theta | z)}$. Our MIA score is $\mathrm{Score}\_\mathrm{MIA}(x; \theta) = \Pr_{z \sim \pi} \big( \mathrm{LR}_{\theta}(x, z) \ge \gamma \big)$.

The term $\Pr(\theta | x )$ is the probability that the algorithm produces the model $\theta$ given that $x$ was in the training set, while the rest of the training set is randomly sampled from the population distribution $\pi$.

Using the Bayes rule, we expand $\mathrm{LR}\_{\theta}(x, z)$ to $\left(\frac{\Pr(x | \theta)}{\Pr(x)}\right)\cdot\left(\frac{\Pr(z | \theta)}{\Pr(z)}\right)^{-1}$ with no approximation.

The term $\Pr(x)$ is defined as the normalizing constant in the Bayes rule, and has to be computed by integrating the numerator of the Bayes rule over all models $\theta'$.

$\Pr(x) = \sum_{\theta'} \Pr(x | \theta') \Pr(\theta') = \sum_{D, \theta'} \Pr(x | \theta', D) \Pr(\theta' | D) \Pr(D) = \sum_{D, \theta'} \Pr(x | \theta') \Pr(\theta' | D) \Pr(D)$

We empirically compute $\Pr(x)$ as the empirical mean of $\Pr(x|\theta')$ by sampling reference models $\theta'$, each trained on random datasets $D$ drawn from the population distribution $\pi$.

We hope the process is more clear. Compared to the previous version, we incorporate the effect of other data points in the training set in the computation of $\Pr(\theta | x )$. Note that, the attack process and our final test remained unchanged.

2. Reference records

Clarification: The percentage refers to the size of the reference models' training set rather than the entire population. So, a portion of the samples used for training reference models is selected as the set of reference records. We have revised the relevant section (Appendix A.8) to accurately reflect this.

Dependence on number of records: All attacks assume they have a large number of reference points, which (except the Home et al and Attack-P in Ye etal 2022 methods) they use all the reference records to train reference models. In our attack, we show that a fraction of these data points should also be used as $z$ reference points in our attack to boost the relativity of the likelihood of observing $\theta$ under $b=0$ versus $b=1$.

3. Selecting $\gamma$ and $\beta$

To compute ROC curve, all methods sweep over values of $\beta$. In our method (as opposed to e.g., Carlini etal 2022), the exact value of each $\beta$ is interpretable and is calibrated: As shown in Figure 14, for $\gamma=1$, the value of $\beta$ is $1-FPR$.

We added Figure 12 (Section A.9) to show the stability regarding $\gamma$. Our attack's performance remains consistent across different values of $\gamma$.

Interpretation of $\gamma$ is that by increasing it, we reduce the attack FP for the same $\beta$. But, if we are sweeping over $\beta$, the ROC curve does not change significantly.

4. Overfitting

We considered diverse benchmark datasets, and also used the Purchase-100, which is a non-image dataset in MIA literature.

To address your comment, we explored the performance of attacks across various ML algorithms. In new Appendix A.6, we analyzed the Gradient Boosting Decision Tree algorithm on Purchase-100.

In new Appendix A.3 and A.4, we analyze the effect of data distribution and architecture shifts (using a different dataset/architecture for reference models than the target model). In all cases, our method consistently outperforms other attacks, and the gap of performance even increases, i.e., other attacks are more susceptible to these changes.

Many thanks for detailed questions and comments.

Weaknesses

1. The presentation of the paper need to be improved. For example, the comparisons between the proposed method and the previous methods needs to be further clarified, especially for the method proposed by Carlini et at., 2022.

We have revised Section 2. The updated version should address this comment. We show that all major MIA methods can be presented within the same structure, as we present in the beginning of Section 2. The major difference between attacks is the way they compute their MIA score. Please see updated Section 2.3 and the newly added Table 1. As we describe in the text, the LR corresponding LiRA has an averaging factor over all $z$, while in our attack we split LR for all $z$, which gives us a better distinguishability of $x$ and non-members. Also, our Bayes computation of LR provides a significant edge over direct computations of LR, as shown in newly added Figures 20 and 21. The new plots in Figure 6 also show the significant power of our attack compared to other attacks.

2. It is unclear why the authors can assume that $\Pr(D | \theta')$ to be a constant.

Appendix A.13 and Figure 22 show that the variance of this quantity is extremely small across different $\theta'$. So, their changes could be ignored without hurting our test power. Also, in the revised explanation of our method, we provide a different yet equivalent and more interpretable derivation of $\Pr(x)$ and $\Pr(z)$, where the effect of $D$ is more intuitive. Note that our computations of LR have not changed.

3. The authors do not test the methods when the model is differentially private.

We have included a new section, Appendix A.5, to assess the impact of using DP-SGD on the performance of attacks with various combinations of DP-SGD’s noise multiplier, clipping norm, and privacy budgets. As shown in Figure 9, our attack consistently outperforms others, especially in more stringent settings.

Questions

1. In Definition 1, you assume a fair coin $b$. What if the probability of the data being a member or not is not $0.5$, and whether your method can be applied to this case?

This is the standard approach for all MIAs. Changing $b$ doesn't change the attacks, but only the interpretation of the TPR-FPR curve.

Something interesting, however, is that if we are concerned with the fact that in practice there are orders of magnitude more non-members than members for any model, we need to pay attention to 2 factors in any MIA: The cost of performing attack on any new data point, and their TPR versus FPR (especially for low FPR and high TPR regions).

2. For the parameters $\beta$ and $\gamma$, how sensitive of these parameters and whether it is hard to find the optimal parameters?

As discussed in the revised Section 2, $\beta$ is common among all attacks, and will be used to generate the ROC curve.

We added Figure 12 (Section A.9) to show the stability regarding $\gamma$. Our attack's performance remains consistent across different values of $\gamma$.

3. What are the standard deviations of your report results? Do you have some confidence intervals in your plots?

Added for Tables 2, 3, 4.

4. For the predication probability functions ($\Pr(x | \theta)$), how sensitive are those hyperparameters?

We conducted a performance sensitivity analysis of our attack with respect to different hyperparameters of computing probabilities. Please see Figure 15. Firstly, as indicated in Table 6 of Section A.10.2, using different functions has a negligible impact on the result of our attack. Furthermore, as illustrated in the figure, our attack's performance remains robust against variations in the soft-margin (m), temperature (T), and higher orders (n).

5. When you compute $\Pr(x)$ for the offline method, what is the computational cost and how the results will be affect by the linear approximations?

Please see new explanations in Appendix A.10.3. We compute $\Pr(x)$ only using the available reference models. If we have 1 or 2 reference models, we use fine-tuning to compute the gap between $\Pr(x)\_{OUT}$ and $\Pr(x)\_{IN}$. Also the new Figure 16 shows that the attack is not very sensitive to small changes to $a$. Notably, overestimating the gap between $\Pr(x)\_{OUT}$ and $\Pr(x)\_{IN}$ has minimal impact.

6. If you continue to increase the reference models, how will your methods look like compared to the method proposed by Carlini et at., 2022?

Please see the new Figure 6. In the offline setting, LiRA Carlini 2022 cannot reach the performance of RMIA and other attacks (e.g., Ye 2022). In the online setting, with a large number of reference models it approaches RMIA. Note that even in that setting, RMIA offline is almost as good as LiRA online (with significantly higher cost of training hundreds of reference models per new MIA query). Please also see Figures 2, 3, 4, 5 for the ROC curves, as we increase the number of reference models.

Many thanks for the questions. We have further improved the clarity of our method in Section 2. The method remains exactly the same, however, we first clearly explain what the structure of the attack and its evaluation is, which is common among all the major MIA methods. Then, we compute our LR which helps computing the MIA score.

1. It is unclear whether the approximation of the likelihood always holds. The idea is that different reference models exhibit similar predictive distributions. Is it assumed that the reference models are trained well and over a large sample size for this assumption to hold? Essentially, having some clarity about the assumptions of the reference models will be useful ...
2. Continuing point 1, how hard is it for the adversary to access/train such reference models and is it a standard assumption in literature? A brief discussion about the adversary's strength will be helpful.

Why reference models? Using reference models (also referred to as shadow models in the literature) is the cornerstone of the SOTA membership inference attacks. Effectively, without reference models, it is very hard to quantify how much inclusion of a given data point $x$ in the training set of a model could have influenced the model. Reference models act as approximations for what the model could have been with or without $x$ in the training set. Without the use of reference models, the MIA attacks would be only able to capture the leakage due to the average generalization gap in the models. With the use of reference models, the MIA attacks are able to capture the leakage due to the memorization of a specific target data point, thus their strength. However, some MIA methods (notably LiRA Carlini et al) require a very large number of models to be trained for each target model, rendering them less useful in practice.

Performance variation of our reference models. We show these results in Appendix A.13. As it is shown, the average probability of population data has a very small variance across different reference models.

Attack performance under low quality reference models. Your comment raises a good point regarding the stability of tests with respect to quality of reference models. Thus, we added new experiments to test this. In Appendix A.3, we demonstrated that training reference models with data distribution shifts (namely, using a completely different dataset than the target model) still yields significantly better results for our proposed attack compared to other attacks. Additionally, in Appendix A.4 (Figure 8), we showed that our attack outperforms other attacks even when reference models are trained with different architectures than the target model. Interestingly, the performance gap between our attack and others increases in such cases. Another important point is that we deliberately avoided taking advantage of overfitted models, as the train-test accuracy gap in our trained model is smaller compared to experiments in prior works. It's worth noting that the performance of attacks naturally increases with overfitted models. As reported in Section 3.1, we trained several models with varying test accuracies, from 67% to 92%, to show the superiority of our attack.

Performance under very few reference models. We demonstrated that even in the extreme case of using only 1 offline reference model, our attack can perform well and completely overcome other attacks, for example achieving at least 28% better AUC and 2x-4x higher TPR at zero FPR across all datasets, compared to the well-known LiRA attack. A comparison of the properties of the MIA test scores assigned by different attacks (as presented in Figure 17-19 in Appendix A.11) highlights that our attack achieves superior separation between scores of members and non-members, particularly when limited to a few reference models.

3. The authors mention that they incur lower costs as they do not have to train models including the target sample. However, does the training of reference models not incur additional costs? Providing a simple cost comparison discussion will help.

Cost of reference models needs to be evaluated along with their added value for the attack outcome. Please see the newly added Figure 6, where we show the AUC of attacks, when we increase the number of reference models.

In the online setting, adversary needs to train $k$ additional models per new MIA query. This is a huge cost. Carlini et al LiRA attack performs poorly for a small to moderate number of reference models, and can achieve a comparable attack performance to ours only when the number of models is significantly large.

In the offline setting, adversary trains $k$ reference models in advance, prior to knowing the MIA queries. Our attack clearly outperforms other attacks, and with a few reference models it performs very close to its best results.

Using a couple of reference models (offline) RMIA outperforms the prior attacks (online or offline)

Thanks for the comments and questions.

Details of the Game and the Test

Missing details: The paper does not provide pseudo code for their suggested attack. Neither is the exact computation of the test statistic described.

• We revised Section 2 thoroughly to better explain the game and different steps of the attack.
• We added Appendix A.1 which includes the pseudo-codes and detailed step-by-step computations needed for the attack.

Membership inference game

The MIA attack setup described in this work is different from the ... previous work ... that does not have the capacity to poison the dataset.

How is the indistinguishability game that you propose required for your attack? Why do you require that your game be different ...?

We have revised the text to clarify this (which we admit was confusing). Our game and the test experiments are exactly the same as that of the prior work, and we do not assume any poisoning. Please see Section 2, where in the beginning of the section we describe the foundation of the test.

Membership Inference Game

Let~$\pi$ be the data distribution, and let $\mathcal{T}$ be the training algorithm.

• The challenger samples a training dataset $S \sim \pi$, and trains a model $\theta \sim \mathcal{T}(S)$.
• The challenger flips a fair coin $b$. If $b = 1$, it randomly samples a data point $x$ from $S$. Otherwise, it samples $x \sim \pi$, such that $x \notin S$. The challenger sends the target model $\theta$ and the target data point $x$ to the adversary.
• The adversary, having access to the distribution over the population data $\pi$, outputs a membership prediction bit $\hat{b} \leftarrow \mathrm{MIA}(x; \theta)$.

Membership Inference Attack

A membership inference attack is a hypothesis testing problem that assigns a membership score $\mathrm{Score}_\mathrm{MIA}(x; \theta)$ to every pair of $(x, \theta)$, and outputs a membership bit through comparing the score with a threshold $\beta$:

$\mathrm{MIA}(x; \theta) = 1 \text{ if } \mathrm{Score}_\mathrm{MIA}(x; \theta) \ge \beta$

For any given threshold $\beta$, the adversary's power, defined as the true positive rate of the attack, and error or the false positive rate, are quantified over numerous repetitions of this experiment.

Our work

We propose a novel method to compute $\mathrm{Score}_\mathrm{MIA}(x; \theta)$. The rest of the framework for the game and the test and their evaluation are the same as the prior work.

Probabilities used in the likelihood ratio

Aren’t p(z) and p(x) just the prior probabilities of observing x and z, respectively?

We have revised the text. $\Pr(x)$ is not the same as $\pi(x)$, which is rather the prior distribution over $x$. The term $\Pr(x)$ is defined as the normalizing constant in the Bayes rule, and has to be computed by integrating the numerator of the Bayes rule over all models $\theta'$.

$\Pr(x) = \sum_{\theta'} \Pr(x | \theta') \Pr(\theta') = \sum_{D, \theta'} \Pr(x | \theta', D) \Pr(\theta' | D) \Pr(D) = \sum_{D, \theta'} \Pr(x | \theta') \Pr(\theta' | D) \Pr(D)$

We compute $\Pr(x)$ as the empirical mean of $\Pr(x|\theta')$ by sampling reference models $\theta'$, each trained on random datasets $D$ drawn from the population distribution $\pi$.

Note in the equation above $\Pr(x | \theta', D)$ is equal to $\Pr(x | \theta')$ because given a model $\theta'$, (the prediction on) a data point is independent of the training set $D$ (i.e., conditional independence between $x$ and $D$ given $\theta'$)

Discussion on the mechanism

The discussion on the mechanism of the author’s proposed attack in section 2.3 ...

We have revised Section 2.3 and added new analysis and discussions. The discussion is around how other methods can be simply constructed from ours via averaging or simplifications (So, we are not deriving any new theoretical results there). Here is our clarification.

MIA is a hypothesis test problem. In our method, we consider the worlds associated with null hypothesis to be the worlds in which $x$ is not in the training set and instead a different data point $z$ is included. This allows us to model the null hypothesis (non-member) worlds in a more fine-grained way, as opposed to the prior SOTA.

In Section 2.3, we also provide a direct way to compute our LR which makes our attack more easily comparable to LiRA.

Our pairwise LR: $\frac{\Pr(f_\theta(x), f_\theta(z) | x)}{\Pr(f_\theta(x), f_\theta(z) | z)}$

LiRA Carlini et al LR: $\frac{\Pr(f_\theta(x) | x)}{\Pr(f_\theta(x) | \bar{x})}$

$\bar{x}$ means the $x$ is not in training set.

We observe that the denominator in LiRA LR is the average case for our LR averaged over all $z$. This reduces the power of LiRA's test. Also, as we show in Appendix A.12, a direct computation of LR requires a large number of reference models. The combination of a pairwise LR and its computation using the Bayesian approach results in higher power and lower cost in our attack.