Real-World Benchmarks Make Membership Inference Attacks Fail on Diffusion Models

We thank the insightful review and would like to address the issue by points:

[W1 & Q1] (difference from overfitting & our contribution) We agree that over-fitting and distribution shifts have been discussed by previous or concurrent works. However, there are differences between over-training of diffusion models and traditional over-fitting. Over-fitting in traditional models could be measured by train-test gaps [C] and is supposed to be a bad phenomena. However, as discussed in L155-L158, over-training (for hundreds of epochs) is necessary to achieve the best FID for small diffusion models. For example, the default training epochs of DDPM on CIFAR-10 is 2048 (=800k*128 / 50k) epochs. Using these default small models is the reason why current MIA evaluation tends to suffer from over-training. However, as shown by our experiments, MIA succeeding on over-trained models may have nothing to do with large-scale models without over-training, e.g. Stable Diffusion, while the latter constitutes the real world concern of MIA. This dilemma is kind of unique for diffusion models. We have mentioned this in our updated new draft in Section 3.

Furthermore, the contribution of this work is more on rectifying current academic practice, that we aim at stopping the wrong trend of depending on flawful evaluation. As mentioned by the reviewer, there are lots of works discussing the over-fitting and distribution shifts in MIA of other models. However, new research in diffusion MIA seems to neglect these discussions and continues to use evaluation with over-training and dataset shifts. For example, one of our concurrent works (link: https://openreview.net/forum?id=LRSspInlN5) still uses DDPM+CIFAR10, DDPM+CIFAR100, and our setup (b) (flawful) as benchmarks. Hence, we believe it is significant and timely to explicitly alert the flaw of current MIA evaluation and call for stopping depending on it. In addition, we provide three prepared and plug-and-play benchmarks, our setup (c) (d) and (e), for realistic MIA evaluation. We do this by searching for all possible large-scale diffusion models with accessible and no-shift members and non-members. This helps future diffusion MIA research adapt swiftly to the real-world setup. We believe both of these two contributions are not trivial.

[W2] (possible adjustments to improve MIA) To provide insights for future adjustments, we have conducted new experiments to demonstrate the correlation between quantified dataset shifts and MIA performance, which validates that with only dataset shifts we can make the MIA fail (Section 5.3). Also, we have updated the analysis of the reason for MIA failure and given brief insights on potential improvements (Appendix A.3.1). To briefly summarize the idea, due to the fact that we only train one step on one specific noise and time step, we cannot distinguish members based on losses with randomly sampled time steps and noises. Instead, we may need to locate the exact time step and noise that the model used for THAT training step.

However, as discussed above, our main goal is to alert the fatal flaw in current diffusion MIA evaluation and provide a realistic benchmark. Designing novel methods is then out-of-the-scope of our primary focus. Hence, we would like to leave this for future work.

[W3 & Q2] (new baselines) We thank the reviewer for the advice and feel sorry for missing these baselines. We have included these works in our references and are currently running experiments on them. However, it is notable that likelihood-based methods [A] perform similarly to loss-based methods and are not considered as baselines in recent MIA research. Additionally, [B] is a quantiled version of GSA in our baselines, which mainly focuses on improving efficiency. Hence, we believe these two baselines will not be exceptions for our conclusion on the diffusion MIA’s failure on real-world benchmarks.

[Q3] (typos) We thank the reviewer and have fixed it in our updated new draft.

Again, we thank the reviewer for the insightful review. If you have further questions, feel free to contact us.

References:

[A] Hu & Pang. Loss and likelihood based membership inference of diffusion models. In International Conference on Information Security. 2023.

[B] Tang et al. Membership inference attacks on diffusion models via quantile regression. International Conference on Machine Learning. 2024.

[C] Carlini et al. Membership inference attacks from first principles. 2022 IEEE Symposium on Security and Privacy. 2022.

Dear Reviewer,

We highly appreciate your constructive reviews that raised the concerns on the contribution of our paper. As we mentioned in the rebuttal, the main contribution of this paper is to alert a wrong trend of the current MIA research on diffusion models. While MIA benchmarks are not necessary to be defective, all existing benchmarks suffer from the two defects we presented and are continuously used in evaluating new methods. Hence, we believe it is necessary to point out the defects and provide a new fair benchmark. We believe our work is significant for the practice of MIA research in belief of the necessity of a fair benchmark. As the discussion phase is about to close, we will sincerely appreciate it if you can engage the discussion and provide further advice on our paper.

[Q4] (implications to promote future research) To provide insights for future adjustments, we have updated the analysis of the reason for MIA failure and given brief insights on potential improvements (Appendix A.3.1). To briefly summarize the idea, due to the fact that we only train one step on one specific noise and time step, we cannot distinguish members based on losses with randomly sampled time steps and noises. Instead, we may need to locate the exact time step and noise that the model used for THAT training step.

However, as discussed above, our main goal is to alert the fatal flaw in current diffusion MIA evaluation and provide a realistic benchmark. Designing novel methods is then out-of-the-scope of our primary focus. Hence, we would like to leave this for future work.

Again, we thank the reviewer for the insightful review. If you have further questions, feel free to contact us.

We thank the insightful review and would like to address the issue by points:

[Q1] (Why our benchmark is realistic) Our benchmark is more realistic than existing benchmarks, because we use models without over-training and non-member datasets with minor or without shifts to member datasets. We would like to summarize the reasons as follows:

Dataset shifts: Membership inference is supposed to only depend on membership. If we use a cat datasets to train a model and use a dog dataset as non-members for evaluation, an image classifier can easily separate these two datasets. However, this is not membership inference, and will immediately fail when we use another hold-out cat dataset as non-members. According to our new experiments in Section 5.1, our datasets do have smaller dataset shifts. Also, Section 5.3 shows that by only tuning the dataset shifts, we can manipulate the evaluation result. This means that current evaluation with big dataset shifts is not fair and realistic.

Over-training: over-training diffusion models for hundreds of epochs on small datasets make the loss of members conspicuously lower than that of non-members. However, real-world security and privacy scenarios mostly involve large-scale diffusion models that are only trained for one epoch on the training dataset. MIA succeeding on over-training models may have nothing to do with these models in the real-world application. Our benchmark uses pre-trained models with only one epoch, which is the minimum training epoch, thus getting rid of over-training.

[Q2 & W1] (in-depth analysis on dataset shift’s impact) We thank the reviewer for the advice and have conducted new experiments to quantitatively analyze how dataset shifts have impact on the MIA performance. Basically, our experiments have two parts:

• Quantifying dataset shifts (Section 5.1). We calculate three distance metrics between member and non-member datasets in our benchmark: normalized Wasserstein distance (NWD), Fréchet Distance (FD), and Mahalanobis Distance (MD). All distances are calculated based on CLIP-large. Among all setups covered, setup (a) and (b) in our benchmark suffer much bigger distances between members and non-members, for example, FDs of 0.32 and 0.24. This validates our conclusion that there are valid shifts between their members and non-members. Therefore, MIA methods could separate these two datasets according to the semantics rather than the membership, which raises the dataset shift concern in the paper. Setup (d) has medium distances between members and non-members, for example, a FD of 0.12. The distances are much smaller for Setup (c) and (e), for example, FDs < 0.10. This experiment quantitatively demonstrates the existence of dataset shifts and shows potential connections between dataset shifts and the MIA performance.

• Relation between dataset shifts and MIA performance (Section 5.3). We construct a series of non-member datasets by mixing our two non-member datasets: COCO-val-2017 (with shifts to LAION) and LAION-MI (no shifts) with different proportions. We pick the proportions by 100% vs 0%, 75% vs 25%, 50% vs 50%, 25% vs 75%, and 0% vs 100%. We evaluate SecMI and PIA on these setups. The result in Figure 1 shows that there is positive correlation between the performance of these two MIA methods and the proportion of shifted non-members data. This shows that one can manipulate the result of MIA evaluation easily by only changing non-members, which is supposed to be irrelevant to the result and that current MIA evaluation is unreliable.

[Q3] (generalization of our conclusion) We notice that there are existing works discussing the failure of MIA on other generative models, e.g. LLMs. However, MIA of LLMs seems to never fall into the hallucination of success as that of diffusion models do. This is because over-training is the default setup of training small diffusion models to achieve the best FID (L155-158 in our new draft). In other words, there is an intrinsic gap between MIA of small diffusion models and that of large-scale diffusion models in the real-world application. We believe this is unique for diffusion models. The main contribution of this work is also to rectify current academic practice in diffusion MIA, to stop the wrong trend of depending on evaluation pipelines far from real-world applications, and to provide a sound and realistic benchmark.

Dear Reviewer,

We highly appreciate your constructive reviews that raised the concerns on the lack of discussion. To address this concern, we have added discussions on the following topics. 1) We quantifying the impact of dataset shifts on MIA performance. The result shows that by only replacing the non-member dataset the MIA performance will suffer a significant drop. Hence, we need to consider the hardest non-member dataset as the real-world setup. 2) We discussion potential directions of future improvement in the area. We believe these two extra discussions can effectively address your concerns. As the discussion phase is about to close, we will sincerely appreciate it if you can engage the discussion and provide further advice on our paper.

We thank the insightful review and would like to address the issue by points:

[1] (x refers to image or image+prompt) x solely refers to the image. We do not consider the prompt because 1) prompts could be highly variable and easily modified in training, thus not being reliable conditions for MIA and 2) None of baseline methods claim strong dependency on prompts. If future methods have this strong dependency, we would like to update our benchmark and take prompts into consideration. Also, we thank the reviewer for the advice and have fixed this in the problem definition.

[2] (Typo in Table 1) Yes, it is a typo. We feel sorry for this typo and have fixed it in our new draft. "LDM + CelebA" has both dataset shift and over-training, for it uses two different datasets as members and non-members and trains the model for 500 epochs on the member dataset.

[3] (in-depth analysis on dataset shift’s impact) We thank the reviewer for the advice and have conducted new experiments to quantitatively analyze how dataset shifts have impact on the MIA performance. Basically, our experiments have two parts:

• Quantifying dataset shifts (Section 5.1). We calculate three distance metrics between member and non-member datasets in our benchmark: normalized Wasserstein distance (NWD), Fréchet Distance (FD), and Mahalanobis Distance (MD). All distances are calculated based on CLIP-large. Among all setups covered, setup (a) and (b) in our benchmark suffer much bigger distances between members and non-members, for example, FDs of 0.32 and 0.24. This validates our conclusion that there are valid shifts between their members and non-members. Therefore, MIA methods could separate these two datasets according to the semantics rather than the membership, which raises the dataset shift concern in the paper. Setup (d) has medium distances between members and non-members, for example, a FD of 0.12. The distances are much smaller for Setup (c) and (e), for example, FDs < 0.10. This experiment quantitatively demonstrates the existence of dataset shifts and shows potential connections between dataset shifts and the MIA performance.

• Relation between dataset shifts and MIA performance (Section 5.3). We construct a series of non-member datasets by mixing our two non-member datasets: COCO-val-2017 (with shifts to LAION) and LAION-MI (no shifts) with different proportions. We pick the proportions by 100% vs 0%, 75% vs 25%, 50% vs 50%, 25% vs 75%, and 0% vs 100%. We evaluate SecMI and PIA on these setups. The result in Figure 1 shows that there is positive correlation between the performance of these two MIA methods and the proportion of shifted non-members data. This shows that one can manipulate the result of MIA evaluation easily by only changing non-members, which is supposed to be irrelevant to the result and that current MIA evaluation is unreliable.

Again, we thank the reviewer for the insightful review. If you have further questions, feel free to contact us.

We thank the insightful review and would like to address the issue by points:

[1] (Must over-training and dataset shifts occur in previous benchmarks?) We thank the reviewer for raising this meaningful question, for which our first draft does not provide a sound explanation. We would like to make a clarification as follows:

First, ticks mean that the benchmark does have the over-training or dataset shifts while crosses mean it does not. There is a typo that "LDM + CelebA" should have both dataset shift and over-training in Table 1 (upper).

Second, it is not a must that an arbitrary benchmark using the model and the dataset in Table 1 (upper) suffers from over-training and dataset shifts. Table is used to show the setups of all existing benchmarks with the two drawbacks. The model and the dataset are considered as parts of the shown setups.

Third, however, while dataset shifts are easy to avoid (by using half of one dataset to train the model and another half as non-members), over-training is difficult to avoid for most small diffusion models. As mentioned in L155, training for hundreds of epochs is necessary to achieve the best FID for small datasets and diffusion models. This may be caused by the limit size of the dataset. Hence, most small diffusion models trained on academic datasets, including CIFAR-10, CelebA, and ImageNet, need over-training to converge. If we pick a small model without over-training, then the model could be unconverged and thus not qualified to benchmark MIA. That is why over-training is hard to avoid when using small diffusion models as MIA benchmarks.

As a result, it is necessary to introduce large-scale diffusion models as real-world MIA benchmarks. These models enjoy the large size of the training dataset so that they could converge with only one epoch. Hence, MIA methods cannot rely on the memorization caused by over-training to easily distinguish members from non-members on these models. This is the main motivation of our work.

[2] (Varied setups) As shown in Table 1, previous benchmarks have included such setups with over-training or dataset shifts. As discussed at L206-L208, these setups are meaningless. MIA benchmarks with dataset shifts can be dominated by image distribution classifiers, which will immediately fail on no-shift benchmarks, while those with over-training turn the problem into over-fitting detection. MIA methods succeeding on these benchmarks do not truly infer membership and cannot be used in real-world scenarios.

The main contribution of our work is that we 1) alert a problematic trend of current MIA research that methods compete on above meaningless benchmarks and 2) provide a practical real-world benchmark for future research to compete on. While we have included two flawful setups a) and b) in CopyMark, we believe there is no need to include more, because this is not relevant to our two contributions.

[3] (in-depth analysis on dataset shift’s impact) We thank the reviewer for the advice and have conducted new experiments to quantitatively analyze how dataset shifts have impact on the MIA performance. Basically, our experiments have two parts:

• Quantifying dataset shifts (Section 5.1). We calculate three distance metrics between member and non-member datasets in our benchmark: normalized Wasserstein distance (NWD), Fréchet Distance (FD), and Mahalanobis Distance (MD). All distances are calculated based on CLIP-large. Among all setups covered, setup (a) and (b) in our benchmark suffer much bigger distances between members and non-members, for example, FDs of 0.32 and 0.24. This validates our conclusion that there are valid shifts between their members and non-members. Therefore, MIA methods could separate these two datasets according to the semantics rather than the membership, which raises the dataset shift concern in the paper. Setup (d) has medium distances between members and non-members, for example, a FD of 0.12. The distances are much smaller for Setup (c) and (e), for example, FDs < 0.10. This experiment quantitatively demonstrates the existence of dataset shifts and shows potential connections between dataset shifts and the MIA performance.

• Relation between dataset shifts and MIA performance (Section 5.3). We construct a series of non-member datasets by mixing our two non-member datasets: COCO-val-2017 (with shifts to LAION) and LAION-MI (no shifts) with different proportions. We pick the proportions by 100% vs 0%, 75% vs 25%, 50% vs 50%, 25% vs 75%, and 0% vs 100%. We evaluate SecMI and PIA on these setups. The result in Figure 1 shows that there is positive correlation between the performance of these two MIA methods and the proportion of shifted non-members data. This shows that one can manipulate the result of MIA evaluation easily by only changing non-members, which is supposed to be irrelevant to the result and that current MIA evaluation is unreliable.

If you have further questions, feel free to contact us.

Dear Reviewer,

We highly appreciate your constructive reviews that raised the concerns on the contribution of our paper. As we mentioned in the rebuttal, the main contribution of this paper is to alert a wrong trend of the current MIA research on diffusion models. While MIA benchmarks are not necessary to be defective, all existing benchmarks suffer from the two defects we presented and are continuously used in evaluating new methods. Hence, we believe it is necessary to point out the defects and provide a new fair benchmark. In addition, we conducted two experiments to quantitatively demonstrate how changing dataset shifts could manipulate the performance of diffusion MIAs. We believe these could effectively address your concerns in the advanced comprehension of dataset shifts. As the discussion phase is about to close, we will sincerely appreciate it if you can engage the discussion and provide further advice on our paper.