We appreciate your review and are happy to see that you found our work to pose a compelling argument for transitioning toward Dataset Inference, and found the paper well-written. We acknowledge your concerns and attempt to respond to them line by line below:

Re: Decontamination of Validation Set

Thank you for your comment. We understand the concern regarding potential contamination in the validation set. To address this, we have conducted additional decontamination using a stricter deduplication method that ensures minimal overlap between the training and validation sets. Specifically, we applied a more rigorous n-gram filtering process to eliminate any shared sequences between these sets. Here are a few key points regarding decontamination:

1. Reference to the Pile Paper: The original Pile paper (Gao et al., 2021) acknowledged challenges with deduplication and contamination between training and validation sets. Despite their efforts, complete decontamination is difficult to achieve, as some overlap in n-grams is inevitable because of domains being identical (for example, arxiv headers).

2. Impact of Contamination on Dataset Inference: If there were any contamination, it would only make dataset inference harder, as the validation set would resemble the training set more closely. Therefore, high n-gram overlap should lead to worse, not better, results. However, our method succeeds in distinguishing between training and validation datasets despite these challenges, indicating the robustness of our approach at solving an even harder problem.

3. N-gram Overlap: We agree with your point about n-gram overlap. A concurrent paper (Duan et. al., 2024) independently found that there is approximately a 30% overlap in n-grams between Wikipedia and arXiv datasets. This study also indicated that non-members with lower n-gram overlap are more distinguishable by existing MIAs. This supports our claim that even with potential contamination and the inherent challenges of deduplication, our dataset inference method remains effective.

Re: Novelty of the Proposed Approach

We appreciate your observation regarding the novelty of our approach. Here are a few key points that highlight the innovative aspects of our method:

1. Achieving a Previously Near-Impossible Goal: We make the previously near-impossible goal of membership inference (MI) achievable. This is a significant feat that opens up new possibilities in the field of detecting training data.

2. Learning which MIAs to Use: Our method involves "learning" which MIAs to use, a critical step that enhances the accuracy and effectiveness of our approach. This was crucial to overcome the biggest technical challenge of our work, that is, most MIAs were worse than random.

3. Operationalizing the Framework: Developing the framework required that we address practical considerations, such as how to handle victims who naturally retain drafts or IID sets of their work.

In summary, the main contribution of our work is not just in the specific techniques but in proposing a comprehensive framework and advocating for a shift towards dataset inference as a field of study ripe for modern day generative models.

Re: Practicality of Obtaining Labeled Validation Data

We acknowledge that obtaining labeled validation data sampled from the same distribution as the test data can be challenging. Indeed, this is the number one open problem that our work creates for future research. That said, a significant part of this work was spent in formalizing a framework. Here are quite a few scenarios where the IID setting should roughly happen in practice:

• Editorial Process: Thousands of NYT articles reach the editor's desk but never get published. These unpublished articles are likely to be more IID compared to two random splits of the Pile dataset.
• Book Drafts: This is also very common with book writers when working on a chapter draft. These drafts would be more IID than random splits of the Pile, in our opinion.

However, we truly need a test of IIDness rather than making claims of the same. This would be a really nice (but also hard) area of research for the future. One strong baseline for the same is the recent “Blind MIAs” work by Das et. al. https://arxiv.org/abs/2406.16201. If any pair of datasets can be distinguished based on these blind membership inference attacks, they should be deemed not IID. Future work can improve upon this.

Some potential directions for bypassing the IID problem in future work also include:

1. Using Synthetic Data: Create IID sets artificially using synthetic data generation techniques.
2. Storing Hashes: Creators may store hashes of IID data at the time of publication to ensure data integrity.
3. Proactive Storage: More proactive work to store generations of artistic work rather than only the final stage.

Once again, we thank you for the constructive feedback on our work. Working on the pointers has helped us improve the quality of our analysis. We look forward to further discussions and improvements in this evolving field.

We appreciate your review and are happy to see that you found our analysis systematic and our paper technically solid with a high impact, along with an overall positive assessment of the work. We acknowledge your concerns and attempt to respond to them line by line below:

Re: Generalization to Commercially Available Models

We recognize the interest in seeing how dataset inference generalizes to commercially available models such as GPT-4 and Claude. Here are a few points that limited the same, but are also great fuel for future work:

1. Lack of Blackbox MIAs: In our work, we leverage grey-box access to the models, that is, we assume that the membership inference attacks have access to the loss values from the underlying model. Since MIAs in LLMs is a relatively nascent field, the research has not yet unlocked successful black-box MIAs, unlike the vision space where we have seen label only MIAs (eg. Choo et. al.). We believe future work can unlock this possibility with API access as well.

2. Access Limitations: Conducting tests with commercially available models is challenging due to limited access to their training data and methodologies. Unlike the Pythia suite, where we had full access to all relevant information, commercial models do not provide the same level of transparency, making it difficult to obtain the ground truth for evaluation. In particular, the presence of ground truth labels for Pythia models allows us to validate that dataset inference is indeed successful in distinguishing members and non-members.

3. Alternative Evaluation Methods: We suggest that future research could explore alternative evaluation methods that do not rely on access to the full training/validation data. For example, researchers could use synthetic data or develop new techniques that infer properties of the training data from the model's outputs. The suggestion of temporally shifted Wikipedia articles would not work in this case because it would give a false sense of success as also noted in concurrent works of Das et. al. and Duan et. al..

Once again, we thank you for the constructive feedback on our work. Working on the pointers has helped us improve the quality of our analysis. We look forward to further discussions and improvements in this evolving field. Please let us know if we can address any remaining concerns.

We appreciate the reviewer's valuable feedback and are happy to hear that they enjoyed reading our paper.

The experiments are conducted on a single series of models, rather than on various models trained with different datasets, algorithms, or even seeds. I think this is a little picky since it's not cheap to retrain a bunch of large models, but I do think it's important for reliable evaluation.

We used the Pythia suite of models, trained on the PILE dataset, since it is unique in providing complete access to both the training dataset and the training procedures. This transparency is crucial as it allows for rigorous and replicable experimentation, ensuring the validity and reliability of our results. Without the information provided along the Pythia suite of models, we cannot reliably know if a point was part of the pretraining set. This could for instance lead us to overestimate the performance of our method, as was the case for prior work. To the best of our knowledge, no other set of LLMs offers the level of accessibility that Pythia models do (except the very recent Olmo model series which only came out after we finished our experimentation). Our work is a call to action for the community to provide more model releases with comparable levels of transparency.

How does the victim choose the suspect set? What if the suspect set is a mix of member and non-member data points? This might happen when the suspect intentionally trains the model on a subset of the dataset.

In general victims can determine suspect sets based on behaviour from a model that is suggestive of having trained on their content. This could include yielding substrings from their novels, or blogs etc. The adulteration of training with only a part of the member data is an interesting consideration. We can reframe the question as follows:

Assume distributions $A$ and $B$ are distinguishable based on a t-test. You can assume they are Gaussian with separate means. Now, an adversary adulterates $A$ with $x\%$ of $B$. Will the t-test succeed in distinguishing the adulterated distribution from $B$?

To determine if the t-test will succeed in distinguishing the adulterated distribution $A'$ (where $A'$ is $A$ mixed with $x\%$ of $B$) from $B$, we need to consider how the adulteration affects the statistical properties of $A'$.

1. Let us assume that the dataset inference scores from stage 3 follow a Gaussian in the original distributions:

   • $A \sim \mathcal{N}(\mu_A, \sigma_A^2)$
   • $B \sim \mathcal{N}(\mu_B, \sigma_B^2)$
   • The t-test can distinguish between $A$ and $B$, implying the means and variances are sufficiently different.

2. Adulterated Distribution:

   • Let $A'$ be the new distribution obtained by mixing $A$ with $x\%$ of $B$. This means $A'$ is a mixture of $(1-x)\%$ of $A$ and $x\%$ of $B$.
   • The mean of $A'$ is:
     $\mu_{A'} = (1 - x) \mu_A + x \mu_B$

3. t-test on $A'$ vs. $B$:

   • For $A'$ and $B$ to be distinguishable, the t-statistic needs to be large enough: $t = \frac{\mu_{A'} - \mu_B}{\sqrt{\frac{\sigma_{A'}^2}{n_{A'}} + \frac{\sigma_B^2}{n_B}}}$

4. Effect of Adulteration:

   • As $x$ increases, $\mu_{A'}$ moves closer to $\mu_B$.
   • For small $x$, $\mu_{A'}$ is still relatively close to $\mu_A$, and the t-test might still distinguish $A'$ from $B$.
   • For large $x$, $\mu_{A'}$ approaches $\mu_B$, making $A'$ and $B$ indistinguishable by the t-test.

The t-test will likely succeed for small $x$ and fail for large $x$. The exact threshold of $x$ depends on the values of $\mu_A$, $\mu_B$, $\sigma_A$, $\sigma_B$, and the sample sizes $n_{A'}$ and $n_B$.

Increasing the sample size $n_{A'}$ can improve the power of the t-test, making it possible to detect smaller differences between $\mu_{A'}$ and $\mu_B$.

Once again, we thank you for the constructive comments. Working on them has helped us improve the quality of our paper. We look forward to further discussions to clarify any concerns that remain.

Thank you for your thoughtful comments and feedback. We are happy to see that you appreciated the method’s robustness and overall quality of the draft. We address the individual concerns below one by one:

Re: Other work studying the failure of MIAs (W1)

We would like to preface this answer by adding that the linked paper was truly a concurrent work to this submission. Crucially, our work not only shows the limitations of MIAs, but goes beyond that by proposing a comprehensive framework to detect if a given dataset (not string) was used in the training of an LLM, a shift that is particularly relevant for modern generative models.
We would like to note that we indeed also duly acknowledged this concurrent work in our submission on lines 187 and 188: “A similar question was concurrently asked by Duan et al. [15], who independently showed that MIAs are only successful because of the temporal shift in such datasets.”

Re: Computation Complexity (W2, Q1, Q3)

The computational complexity of the method is relatively low, and roughly equivalent to the computation cost of performing each MIA required in our work.

1. For instance, considering the Min-K% MIA, this would mean performing a single forward pass on each example, and aggregating the token-wise loss to calculate the final metric score. Other methods are perturbation-based or reference model-based, requiring two forward passes.
2. In the attached code, we provide detailed method to perform each of the 50 MIAs we performed in metrics.py. Many of the MIAs are dependent on each other, for example, all the Min-K% and Max-K% MIAs can be computed in a single forward pass. Overall, in practice we only require 17 forward passes to perform all the attacks.
3. Given that most datasets require less than 500 examples (of train and val) to perform dataset inference. This would mean performing on the order of 1K forward passes. In our experiments, each forward pass is of context length 1024. But can change depending on the model configuration.
4. Finally, most of this computation can be batched to perform many forward passes together. Given modern day hardware, this cost is low enough that it can be performed in about 4 hours of an A6000 GPU node even for models like pythia-12B (for one dataset).
5. The final cost of “learning” which MIAs to use is almost negligible, as it just requires to fit a 50 dimensional linear layer to a few values (can be done on CPUs in less than 30 seconds). To conclude, the method is indeed feasible for model with more parameters. We tested on models from the Pythia suite, where the biggest one has 12B parameters but the method can be easily run for models with 70B parameters (like Llama 2) or larger.

Limitations of Pile Dataset + Pythia

We agree that repeating the experiments on more models and datasets would lead to stronger empirical supports. When it comes to open-weights + open-source pretrained models released online, to the best of our knowledge, the Pythia suite of models is the only one releasing complete information about its training + validation dataset (along with the more recent Olmo model series which only came out after we finished our experimentation). Without this information, we cannot reliably know if a point was part of the pretraining set and thus cannot reliably validate the proposed method. We would like to emphasize that even though the Pythia suite and Pile Dataset may seem like one single entity, this is actually a collection of 4 models x 20 data subsets, that offers significant generality to our method, much beyond any other random split of publicly available data might offer.

Limitation of application to datasets (and not single strings of text)

We believe that in the modern landscape of generative AI, victims naturally contain multiple sequences of text on the internet. For instance, an article in the New York Times would be composed of multiple sequences of context length 1024. Similarly, the same organization, like NYT has multiple articles that it together wants to own copyright over. Victims who will file legal suits will likely always possess such characteristics. Hence, it is quite natural that we are in a scenario where individual sequences no longer exist in isolation (unlike past classification datasets in the vision space). This setting is also relevant to creative artists having many photographs, and paintings.

Existence of IID validation dataset

We acknowledge that obtaining labeled validation data sampled from the same distribution as the test data can be challenging. Indeed, this is the number one open problem that our work creates for future research. That said, a significant part of this work was spent in formalizing a framework. Here are quite a few scenarios where the IID setting should roughly happen in practice:

• Editorial Process: Thousands of NYT articles reach the editor's desk but never get published. These unpublished articles are likely to be more IID compared to two random splits of the Pile dataset.
• Book Drafts: This is also very common with book writers when working on a chapter draft. These drafts would be more IID than random splits of the Pile, in our opinion.

Some potential directions for bypassing the IID problem in future work also include:

1. Using Synthetic Data: Create IID sets artificially using synthetic data generation techniques.
2. Storing Hashes: Creators may store hashes of IID data at the time of publication to ensure data integrity.
3. Proactive Storage: More proactive work to store generations of artistic work rather than only the final stage.

Once again, we thank you for the constructive feedback on our work. Working on the pointers has helped us improve the quality of our analysis. We look forward to resolving any remaining concerns during the rebuttal-response period.

Thank you for your feedback and comments!

W1, Q2: Existence of IID validation dataset

We acknowledge that obtaining labeled validation data sampled from the same distribution as the test data can be challenging. Indeed, this is the number one open problem that our work creates for future research. That said, a significant part of this work was spent in formalizing a framework. Here are quite a few scenarios where the IID setting should roughly happen in practice:

• Editorial Process: Thousands of NYT articles reach the editor's desk but never get published. These unpublished articles are likely to be more IID compared to two random splits of the Pile dataset.
• Book Drafts: This is also very common with book writers when working on a chapter draft. These drafts would be more IID than random splits of the Pile, in our opinion. As the Reviewer raised in Q2, independent chapters of books may not be IID, but their progression will likely yield an IID set. However, this is up to debate and in practice we need an IID verifier (in future work) as opposed to arguing about the same. One strong baseline for the same is the recent “Blind MIAs” work by Das et. al. https://arxiv.org/abs/2406.16201. If any pair of datasets can be distinguished based on these blind membership inference attacks, they should be deemed not IID. Future work can improve upon this.

At the same time, some potential directions for bypassing the IID problem in future work also include:

1. Using Synthetic Data: Create IID sets artificially using synthetic data generation techniques.
2. Storing Hashes: Creators may store hashes of IID data at the time of publication to ensure data integrity.
3. Proactive Storage: More proactive work to store generations of artistic work rather than only the final stage.

W2: Requirements on amount of data

In practice, the size of the training and validation datasets can be exactly equal to the size of the dataset used for testing the attack. This means that even if a potential victim has only 1000 sequences of 1024 length context of text data that was trained on (along with an equivalently sized validation set), this is a sufficient condition for performing dataset inference. As shown in Figure 6, the number of examples needed for dataset inference to be successful decreases with model size and also depends on the dataset, which means we may require less than 1000 sequences in practice.

W3: The method is tested on the single model's family

The Pythia suite of models, trained on the PILE dataset, is unique in providing complete access to both the training dataset and the training procedures. This transparency is crucial as it allows for rigorous and replicable experimentation, ensuring the validity and reliability of our results. Without the information provided along the Pythia suite of models, we cannot reliably know if a point was part of the pretraining set. This could for instance lead us to overestimate the performance of our method, as was the case for prior work. To the best of our knowledge, no other set of LLMs offers the level of accessibility that Pythia models do. Our work is a call to action for the community to provide more model releases with comparable levels of transparency.

W3: and the efficiency is demonstrated only on the largest model of the family

We also demonstrated the efficiency of our method on other models from the suite, in addition to the largest one. Please refer to Figure 10 in the Appendix, which is an extended version of Figure 6a (from the main paper) with different model sizes. An interesting observation is that the curves for median p-values and Wikipedia’s p-value stay similar with respect to models of different sizes. However, this is not the case for the max p-value curve. This indicates that dataset inference does not rely on a large number of data points or model parameters for most datasets, whereas they may be necessary for some particular datasets which smaller models do not learn well.

W4: Less clean data and less transparent model

For less transparent models, there is no ground truth to determine whether a specific data point was used in training or not. Even if we hypothesize that a model is trained on a particular data source, such as Wikipedia, we lack access to an unseen IID validation set. Note that simply collecting Wikipedia articles published after the model’s release is insufficient due to the temporal shift in concepts. Since this is the main limitation of prior work, we opted to study the Pythia suite of models in our experiments.

Q1: Other MIAs

We fully agree that the top-k% MIA is not representative enough to claim the failure of all MIAs. This is why we tested all MIAs we could find including the perturbation-based ones and reported these results in Figure 7 -- we apologize that due to the large amount of MIAs we tested this figure cannot fit in the main body of the manuscript, we will either add it to the main paper (using the additional 1 page if accepted) or try to and emphasize the reference to the Figure in the main body.

Q3: Features used in dataset inference

We performed an ablation study for the features used in dataset inference and the results are shown in Figure 5(a) with a more detailed version in Figure 8 in the Appendix. We did not find that any particular features were most informative across all datasets. Instead, feature informativeness is highly dataset dependent--most of the features contribute positively for some datasets but negatively for others.

Once again, we thank the reviewer for the constructive feedback on our work. Working on the feedback has helped us improve the quality of our work. We look forward to further discussions to clarify any concerns that remain.

Thank you for the thoughtful review and positive feedback.

W1: Absence of hyperparameter gammas.

We measure the success rate of MIAs using the Area Under the Curve (AUC) metric. The AUC metric is advantageous because it provides a comprehensive measure of the model's performance that is independent of the parameter $\gamma$. This independence ensures that our evaluation is robust and not influenced by the specific choice of $\gamma$. As far as dataset inference is considered, it only uses the score given by each MIA, and automatically determines how to weigh each MIA in Stage 2 as described in the paper, hence once again not needing the same parameter.

W2: Explore datasets beyond the Pile dataset and other models except Pythia.

The Pythia suite of models, trained on the PILE dataset, is unique in providing complete access to both the training dataset and the training procedures. This transparency is crucial as it allows for rigorous and replicable experimentation, ensuring the validity and reliability of our results. Without the information provided along the Pythia suite of models, we cannot reliably know if a point was part of the pretraining set. This could for instance lead us to overestimate the performance of our method, as was the case for prior work. To the best of our knowledge, no other set of LLMs offers the level of accessibility that Pythia models do. Our work is a call to action for the community to provide more model releases with comparable levels of transparency.

W3: The title is too broad. The paper proposes a mixture of MIAs.

While dataset inference builds upon membership inference, it argues for a new paradigm of privacy research where we consider the membership of an entire dataset rather than individual text sequences. To clarify our stance, we are only using a mixture of MIAs to perform the task of dataset inference. Even when each MIA performs close to (and often worse than) random, dataset inference can tease out statistical signals from weak membership attacks.

W4: Qualitative analysis of low and high p-values?

Our p-values for true positive cases are significantly below the set threshold of 0.1, while the p-values for true negative cases are substantially above this threshold. However, p-values alone are rather not suitable for qualitative analysis due to their nature as a probabilistic measure rather than a direct indication of effect size or practical significance. Instead, we can use the number of data points required for our LLM dataset inference method. We observe that structured and well-formed datasets, such as Wikipedia, require fewer data points (i.e., 50) for effective inference compared to less structured datasets like Pile-CC (which requires 300 data points), which contain raw web pages. The table below presents the number of data points required for our method per dataset on the Pythia-12b model.

Q1: How did you select the 52 MIA metrics?

We incorporated all available MIAs at the time of developing our method and further extended them to extract additional features, thereby capturing more information and enhancing the success of dataset inference. For instance, rather than using a single K parameter for the MinK-Prob MIA [32], we employed multiple values of K. We performed an ablation study for the features used in dataset inference, and the results are shown in Figure 5(a) with a more detailed version in Figure 8 in the Appendix. We did not find that any particular features were most informative across all datasets. Instead, feature informativeness is highly dataset-dependent--most of the features contribute positively for some datasets but negatively for others. However, we do see some redundancy between many of the Min-k and Max-k% MIAs, hence, they can be ignored if required without a performance tradeoff. That said, there is no computation overhead in computing all of them, as they can be computed in one single forward pass of the model.

Q2: In Figure 6 (b), training set deduplication results, why does the violin plot distribute like sandglass?

The main purpose of Figure 6 (b) was to observe whether the larger models experience more memorization, and how this can allow performing dataset inference more reliably. We would refer you to Figure 10 in the Appendix which provides a better breakdown of how different model sizes perform for dataset inference. Once again, when considering p-values, since it is a statistical test, we generally do not look at the qualitative distribution of the same, and only their binary position below or above the chosen significance threshold. Overall this suggests that for smaller models, there may be some false negatives (because they do not learn the distribution),

Describe dedup and non-dedup settings.

We tried to explain the setup for the experiment in the description of Figure 6 (b) and lines 331-332. However, we will rewrite it to clearly state that: Deduped denotes a version of the Pile dataset where the documents are deduplicated within, and across the data splits (train/test/validation). Non-Deduped is the original version of Pile without any deduplication.

Working on the feedback helped us improve the quality of our work. We look forward to further discussions to clarify any concerns that remain.