Blind Baselines Beat Membership Inference Attacks for Foundation Models

Thank you for your detailed feedback and for taking the time to review our work. Unfortunately, it seems that the reviewer has misunderstood the core contribution of our work. We hope the response below can clarify what our work does.

---

The assumption of this paper on "blind" is not correct. "Blind" should be on both the model and data set [2021], but this paper relies on too much target dataset information.

The referenced paper uses “blind” in a different sense than ours. In Hui et al., a blind attack is one that only has black-box access to the model.

We consider attacks that do not have any access to the model at all! Such an attack shouldn’t even be able to do membership inference since it can’t possibly infer any information from the model. So in principle it should be irrelevant how much information our attacks have about the target dataset. And yet we show that such “blind” attacks can still distinguish “members” and “non-members” because the evaluation datasets are badly constructed.

We will clarify this terminology in our paper.

The paper provides limited experimental details. For instance, it does not specify which models were targeted for the membership inference attacks.

This is a misunderstanding of our paper. Our attacks do not use any access to a model—this is a core point of our entire work.

The paper proposes ideas for constructing better datasets for evaluating membership inference attacks, but it does not provide experimental results or analysis on whether the blind attack would still outperform SOTA methods on these improved datasets.

By definition, a dataset with an IID split would not allow any blind attack to work better than random chance. For completeness, we verified this on the PILE, and found that our blind attacks indeed cannot distinguish the train and test sets better than random.

Current membership inference attacks are typically evaluated across multiple datasets. For example, Zhang et al. [2024a] evaluate their Min-K%++ attack on Wikipedia, GitHub, Pile CC, PubMed Central, and many other datasets to demonstrate generalizability. However, the blind attack’s performance on other datasets is not explored in the paper, making it difficult to conclude that current evaluations are entirely flawed based on the results from just one dataset.

Indeed, we are not claiming that existing evaluations are completely flawed. We are just saying that the use of the specific datasets we consider should be avoided because these datasets do not provide any meaningful signal.

So, for example, if a paper evaluated their MI attack on WikiMIA and the Pile, we argue it would be better if they just evaluated on the Pile.

Also note that we are not claiming that any individual attack (such as Min-K%++) is flawed. We are saying that (parts of) the evaluations of these attacks are flawed. It is entirely possible that the attacks are very effective, but the current evaluations cannot be trusted to show this.

Duan et al. (2024) propose that temporal shifts can influence the performance of membership inference attacks. Could you elaborate further on the differences between your work and Duan et al. [2024]?

Past works have indeed pointed out issues of distribution shifts for MI evaluations. However, these works have all focused on one specific type of shift present in one or two datasets (temporal wiki and arxiv).

We go further and present a systematic analysis of biased members and non-members across 9 published MI evaluation datasets of three different types. We show with experiments that the shift is so severe, that any MI evaluation scores on these datasets cannot be trusted. We also show, more alarmingly, that this issue pervades other kinds of datasets - specifically our other two types of dataset constructions (biases in data replication, distinguishable tails) that are not constructed based on a temporal split.

We will clarify our contributions and relationship to prior work.

Thank you for your detailed feedback and for taking the time to review our work. We understand that the significance of our findings may not have been fully apparent, and we appreciate the opportunity to clarify the key contributions of our paper.

---

The comparison experiment setup is unclear. In Section 3, the authors first extract all dates present in the text and then proceed with date detection attacks. Are samples lacking dates excluded from your inference attacks?

For samples that do not contain a date, our attack simply has to revert to random guessing (or, to minimize false positives, we simply output “not member”). So we don’t do any special filtering here. We apply our attack (and prior ones) to the full dataset.

In your date detection attack, could you elaborate on how the date threshold is selected?

Since we know the dataset and how it was constructed, we simply select the dataset’s cutoff date as our threshold date. More generally, we could also iterate over a few choices of dates and choose the best one based on a validation set.

The core of this paper is to point out the shortages of existing MI attacks on foundation models. However, in the introduction, the discussion does not revolve around this point but rather focuses on how simple attacks can also achieve good results.

Our point in the introduction was to say that since blind (“naive”) attacks can achieve high scores, existing MI evaluations are flawed. We will clarify this.

The experimental section is divided into sections based on the datasets, which makes it difficult to correspond with the previously mentioned common reasons for the intrinsic differences.

We do break down the evaluation section into subheadings 4.1, 4.2, and 4.3, each corresponding to a particular reason for differences as the reviewer suggests. Within each of these subsections, we have one sub-subsection per defense. Does the reviewer have a different structure in mind that would be easier to follow?

Regarding your Bag-of-words classification attack, what is the underlying insight or motivation for this approach? Why can different word combinations be used to infer membership properties, particularly without any interaction with the foundational models?

The motivation is basically a generalization of our date-extraction attack. Since the members and non-members come from different distributions, there are likely some words that are more likely in one distribution than the other, that can be used to make a good guess.

This paper primarily concentrates on the datasets used for MI evaluation, but it does not account for the influence of foundational models. However, the objective of MI attacks is to discern differences between a model's behavior on members and non-members. Even with datasets that are easily distinguishable based on dates or words, existing attacks still fail to achieve satisfactory results after considering the foundational models. Does this suggest that the current use of these datasets remains valid? Furthermore, by focusing on direct distinctions in membership labels and disregarding interactions with foundational models, are the signals you use for differentiation, such as dates and words, being utilized by existing MI attacks that do interact with foundational models? If current MI attacks are not leveraging your information, does it further indicate that the use of these datasets is reasonable (at least at current stage)?

This is a good question. Current MI attacks indeed perform very poorly even on very biased datasets (worse than our blind baselines). While this suggests there is room for improvement, we would still argue for these datasets to be dropped. Indeed, if a new attack comes along that does much better on these datasets, we wouldn’t know if this is because the attack actually is better at extracting membership signal from the model, or if it is just picking up the biased features that our attacks rely on.

Thank you for your detailed feedback and for taking the time to review our work. We understand that the significance of our findings may not have been fully apparent, and we appreciate the opportunity to clarify the key contributions of our paper.

---

The authors claim that current state-of-the-art MIAs fail to extract meaningful membership information, relying only on biased dataset evaluation results. However, this assertion may be overstated

This is not the claim we make. We claim that the performance of state-of-the-art MIAs cannot be distinguished (or is even outperformed) by naive baselines without access to the target model. Our baselines indeed use information that an MI attack might have a hard time extracting. But the point here is that an attack could, in principle, use such features which give no information about the actual ability to extract membership signal.

As a path forward, the paper advocates for future MIA evaluations using PILE, DataComp, or DataComp-LM. However, it is unclear whether these datasets also suffer from distribution shift issues.

Since the train and test sets of these datasets were selected IID, we are guaranteed that no distribution shift exists. For completeness, we tested our attack strategies on the train/test split of the PILE, and got an advantage over random guessing that was not statistically significant.

To ensure robustness of the evaluations, it would be beneficial to repeat the experiments with different random dataset splits, recording the mean and variance of attack success rates.

The results reported are averaged over repeated experiments (10-fold cross validation as mentioned on line 197) with random splits - in the cases where the blind attacks need a “training” phase - such as the bag of words attack and the greedy rare word selection attack.

Date detection based blind attack does not require any “training data” so the entire dataset is used as the test data. Since there is no randomness in this method: neither in the train-test split nor in the actual attack method, this experiment is not repeated and cross-validated. Any repetition will give the same score.

The novelty and technical contributions of this paper appear incremental. Distribution shift issues in evaluation datasets have been previously discussed by Duan et al. and Maini et al.

Duan et al. and Maini et al. focus on temporal shifts within a single dataset (WikiMIA), whereas we provide a systematic analysis of biased members and non-members across three types of biases in nine published MI evaluation datasets. In contrast to these works, we also don’t merely show that a distribution shift exists, but that it is large enough to invalidate all MI evaluations conducted on these datasets to date.

We believe this broader perspective deserves greater attention in the community, as we show that issues in MI dataset creation are not a one-off event, but a systematic issue that plagues the entire field. And yet, many of these datasets are still being used for evaluating MI attacks (as a timely example, ref [3] below which was recently awarded an Outstanding Paper Award at EMNLP proposes a new MIA method and evaluates it on WikiMIA and BookMIA—datasets that we show are severely biased). So can we truly trust these reported numbers? It is crucial for researchers to approach evaluation with caution to avoid producing and propagating misleading results.

We believe that publishing this work in a venue such as ICLR, which plays a pivotal role in shaping the trajectory of machine learning research, will help ensure these concerns are brought to the forefront of the community's attention.

How do state-of-the-art attacks perform relative to blind attacks on unbiased datasets?

On a truly unbiased dataset, our blind attacks would perform no better than chance (by definition). Prior work (e.g., Duan et al. 2024) show that current MIAs also do essentially random guessing on such datasets, but it is possible that future, stronger attacks could extract meaningful signal from unbiased datasets.

[1] Do Membership Inference Attacks Work on Large Language Models? ICLR 2024.

[2] LLM Dataset Inference: Did you train on my dataset? Arxiv,2406.06443.

[3] Pretraining Data Detection for Large Language Models: A Divergence-based Calibration Method. Arxiv:2409.14781.

Thank you for your detailed feedback and for taking the time to review our work. We understand that the significance of our findings may not have been fully apparent, and we appreciate the opportunity to clarify the key contributions of our paper.

---

The authors' framing of certain works as "concurrent" appears to minimize substantial overlaps, particularly with [1] and [2] which preceded the ICLR deadline by 4 and 8 months respectively.

It was not our intent to minimize overlaps with these works. Our work was originally released on arxiv two weeks after [1], but this is of course a while ago now.

Regarding overlap, we acknowledge that past works have pointed out issues of distribution shifts for MI evaluations. However, these works have all focused on one specific type of shift present in one or two datasets (temporal wiki and arxiv). We go further and present a systematic analysis of biased members and non-members across 9 published MI evaluation datasets of three different types. We show with experiments that the shift is so severe, that any MI evaluation scores on these datasets cannot be trusted. We also show, more alarmingly, that this issue pervades other kinds of datasets - specifically our other two types of dataset constructions (biases in data replication, distinguishable tails) that are not constructed based on a temporal split.

We will clarify our contributions and relationship to prior work.

The conclusion (L413) takes a problematic stance by seemingly absolving model trainers of accountability.

We do not intend to suggest that model trainers should be absolved of any kind of accountability. It is not clear to us where our paper suggests this. Our point is mainly that performing MI evaluations as they are done today provides no signal. What we meant to say in L413 is that if researchers want to show they have a strong MI attack, they cannot use mostfoundation models to demonstrate this (unless we find a better way to construct member and non-member datasets).

The claim on L464-465 about "no knowledge of the model" requires clarification.

A MIA necessarily requires the attacker or auditor to be able to interact with the target model. In contrast, our blind attack directly distinguishes members and non-members based on intrinsic differences in features.

L101: "Many of these" - Please be exact. Consider adding a table that talks about MI attack attempts on foundation models, the benchmarks they use (propose or new), and whether they suffer from the train-test split issues that the authors mention here.

Please refer to Section 4 and Table 2, which provide a detailed report on the most effective MI attacks on foundation models. All the datasets listed in the table encounter issues related to train-test splits, as evidenced by the high scores achieved by the blind attack.

L154: The assumption about future dates is oversimplified and overlooks legitimate cases in fiction, climate research, and policy documents

Yes, that’s why we say it is a heuristic. Our goal here is to create a blind MI attack with high TPR at low FPR. This heuristic is enough to do this (although it does have some false-positives as the reviewer suggests). It is not clear to us why this is an issue?

Some of the "blind" baselines rely on detecting data references like 2024 etc. As an adversary cognizant of the training cutoff (or even an auditor), a simple solution to fix this shift would be arbitrarily replacing all such data references for non-members

The point of our blind attacks is not to obtain the highest possible performance on membership inference in a robust manner. The point of our blind attacks is to highlight that evaluation datasets that are popularly being used are not suitable for evaluating membership inference attacks. These edits would indeed make the specific blind baseline we consider weaker, but it would not fix all distribution shifts and so a different blind baseline would likely still work.