Unlocking Post-hoc Dataset Inference with Synthetic Data

The limited model scale and dataset size constrain the generalizability of the conclusions.

We performed additional experiments on larger Pythia models (2.8B, 6.9B). Moreover, we present results on Olmo 7B, which is trained on a very large training corpus of 4.5TB. The results show our method can scale to much larger models and dataset sizes. Please refer to our response to Reviewer z1aa (https://openreview.net/forum?id=a5Kgv47d2e&noteId=z18yFkoW8x) for detailed results.

The justification for hypothesis testing, particularly Equation (3), lacks sufficient reasoning. [...] Specifically, what is the range of values for c(⋅)? Discrete or continuous?

The difference comparison t-test in Eq. 3 is designed following our proposed suffix completion generation task. In the suffix completion task, the suspect and generated heldout suffixes have the same position within a sentence, making the two suffixes directly comparable. To this end, we propose to quantify the distributional gap in each suspect-heldout suffix pair. The gap is measured with only text signals (by the text classifier), and with both text and MIA signals (by the combined classifier). If the gaps in suspect-heldout pairs are larger with extra MIA signals, then the suspect suffixes are likely to be used for training.

Specifically, the values for c(⋅) are continuous, ranging from 0 to 1.

There appears to be a contradiction between the first paragraph of Section 2.2 and Equation (1).

We modified the text in Section 2.2 as follows: “The null hypothesis is that the average scores for $D_{sus}$ are higher than or equal to those for $D_{val}$.”

The fine-tuning setup is narrow (only one recipe is tested)

We performed additional experiments under two more fine-tuning setups, (1) finetuning with LoRA for 10 epochs and (2) full-finetuning, as follows.

Our results suggest that, with more iterations or larger parameter size, the membership signal is stronger and therefore easier to detect. This means that the setup in our submission is the most challenging one.

Only two specific classifiers are tested without exploring stronger alternatives.

We explored a few stronger text classifier backbones and chose the simple 2-layer GPT2-based classifier. Considering the limited number of tokens provided by the author in the DI scenario, stronger text classifiers can be easily overfitted, especially for SoTA LLM-based text classifiers. Here, we present the results for different architectures with different parameter sizes.

The results show that the simple GPT2-based classifier ((initialized+2 layers) can achieve the best AUC. Additionally, this simple classifier has a significantly shorter training time, making the method more practical when faced with more queries.

Typo: Line 283 and Line 310

We have modified those texts in our manuscript.

Can you provide a more detailed explanation of the rationale in Section 4.3? A concrete example would be helpful.

We added a more detailed explanation with an example in the manuscript for better understanding. We also added a figure to visually demonstrate the idea.

Can your method defend against a malicious actor?

The existence of a malicious actor is one of the important problems we would like to address for the previous LLM Dataset Inference—the author might provide a held-out set with a distributional shift to trigger a false positive result. With our approach, the arbitrator can synthesize a reliable held-out set to avoid false positives caused by a malicious user. We think this should be a standard pipeline for dataset inference and visualize it in Figure 1.

In the ablation study, you included ICL paraphrasing. I am curious about the results when using preference optimization for generation.

We extended Table 5 and provided extra results of three baselines (preference optimization, ICL text completion, ICL paraphrasing) together with two more ablation studies for each component for our generation method. Please refer to the second table in our response to reviewer hUkh (https://openreview.net/forum?id=a5Kgv47d2e&noteId=aEzlhvnQm9) for more detailed results.

We thank the Reviewer for the positive feedback and encouraging comments.

However, it may struggle with highly specialized datasets (e.g., medical, legal, or technical texts) where domain-specific patterns might not be captured. This could lead to larger distributional shifts and reduced reliability in these contexts.

We would like to note that the Pile is not a homogeneous dataset and has many different data subsets from various domains, formats, and languages. For example, the PubMed Central (PMC) is a subset of the PubMed online repository for biomedical articles (medical). The FreeLaw subset contains opinions from federal and state courts (legal). The ArXiv subset contains technical text mostly from papers in the fields of Math, Computer Science, and Physics. EuroParl is a multilinguall corpus consisting of the proceedings of the European Parliament in 21 European languages. StackExchange contains user-posted content on the Stack Exchange network in the question-answer format covering a wide range of subjects. We present examples of these synthetic texts from the different subsets of the Pile in the Appendix E. These samples show the diversity of the subsets in the Pile. Here, we also demonstrate that our method generalizes well across different pile subsets and model parameter sizes.

We further extend our analysis beyond the Pile and test our method on the Dolma dataset, which was built from a diverse mixture of web content, scientific papers, code, public-domain books, social media, and encyclopedic materials. We run additional experiments using Olmo 7B, with the member data from Dolma-v1_7 and the non-member data from Paloma. We observe that our method is also effective for the different data types. The results are provided in the table below (following Table 4 in our submission):

Is the result in table 5 only for the single-author dataset? Do similar trends also hold for the Pile dataset?

The results in Table 5 are for the Pile. Below, we also show that our data generation procedure is necessary for the single-author dataset.

The results for single-author data follow a similar trend as the Pile, showing the importance of each component in our framework.

The authors do not mention what blog posts they use. Could this be clarified?

The blog posts are from the financial domain, regarding stock investment suggestions. Due to copyright reasons, the blog post data is not available for sharing yet.

Additional model sizes (2B, 7B) would help illustrate the scaling behavior.

We provide the results to demonstrate the effectiveness of our method across larger model sizes of Pythia (2.8B and 6.9B) and on different model architecture (Olmo 7B). Please refer to our response to Reviewer z1aa (https://openreview.net/forum?id=a5Kgv47d2e&noteId=z18yFkoW8x) for detailed results.

Another simple data generation baseline, which should be considered, is an in-context suffix completion approach.

We performed this baseline in Appendix A.1 “Text Completion Prompt”. Concretely, as an approach of in-context learning, we give ten examples from the documents, and prompt the GPT-4-Turbo model to finish an incomplete document. This baseline performs slightly worse than ICL paraphrasing, and the GPT2 classifier can easily detect the distributional shift with an AUC of 99.2%.

How critical is the synthetic data generation approach given contribution (2)?

Besides the results in Table 2 and Appendix A, we also extended the following results in Table 5 to show the importance of our data generation approach. Concretely, we show that all the generation baselines do not work even with our self-calibration methods (Row 1-6). Moreover, we show that each component in our generation procedure is critical to our framework (row 7-10).

should we use distribution testing to ensure that the suspect and held-out set are statistically similar as a standard part of the DI protocol?

Adding distributional testing as part of DI protocol only strengthens its robustness but not utility. When a distributional shift is identified, it is still hard for the author to provide held-out data required for DI (as shown in Sec. 3). This disables further DI procedures, even when there is a copyright infringement. This is exactly the problem we would like to address in this work—instead of asking the author to provide statistically similar held-out data, the arbitrator should be able to synthesize a reliable held-out set. We think this should be a standard protocol for dataset inference and visualize it in Figure 1.

"The null hypothesis is that the average scores for $D_{sus}$ is lower than for $D_{val}$" Shouldn't the null hypothesis (eq 1) be that the scores for the two distributions are the same?

The text should be “The alternative hypothesis is that the average scores for $D_{sus}$ is lower than for $D_{val}$”. If $D_{sus}$ is trained on the target model, then its MIA scores (e.g. loss) should be lower than the IID held-out set $D_{val}$. The null hypothesis, against the alternative hypothesis, is that “the average scores for $D_{sus}$ is higher than or the same as those for $D_{val}$”. This also aligns with Eq. 1.

Does the inferred column in Table 4 need to be updated to indicate some of the errors, for example in the case of Ubuntu IRC it looks like there is a false positive?

As pointed out in Sec. 5.3, although the difference comparison t-test works for all subsets, the AUC comparison t-test fails for three subsets: Ubuntu IRC, PubMed Central, and PhilPapers. Therefore, we suggest the difference comparison t-test as a better metric. We have also modified the table accordingly.

Suggestions for figures and symbols

We thank the suggestions from the reviewer, and made changes as suggested.

The evaluation might be strengthened by testing on more recent LLM architectures and diverse model sizes.

We provide results to demonstrate the effectiveness of our method across larger model sizes of Pythia (2.8B and 6.9B) and different model architecture (Olmo 7B). Please refer to our response to Reviewer z1aa (https://openreview.net/forum?id=a5Kgv47d2e&noteId=z18yFkoW8x) for detailed results.

Weakness 1 & Question 1: How does your approach for synthesizing held-out data [...] differ from prior methods for synthetic data generation?

The goal of our work is not to propose a novel method for synthetic data generation. Instead, our main contributions lie in combining high-quality and nearly IID synthetic data generation with post-hoc calibration within a framework of dataset inference to provide a practical tool for real-world litigations.

Moreover, we note that directly applying previous generation methods only have limited performance for our task (shown in Table 2). Therefore, we introduce two simple but effective designs tailored for our framework:

(1) Segmenting and shuffling author-provided documents: The segmented and shuffled sequences constructs IID generator training and inference splits, facilitating easier synthesis by the data generator.

(2) Suffix comparison scheme: Both the suspect and synthetic suffixes share a common prefix, which ensures the two suffixes have the same position within a sentence. This enables direct comparison between original-synthetic suffix pairs, which also supports the difference comparison t-test in our post-hoc calibration.

We present the following results to demonstrate the importance of the two key designs in our held-out data synthesis:

With the two proposed designs, our method has the lowest AUC_Text, indicating it can synthesize nearly IID held-out data. Moreover, the generated held-out set has a significantly higher AUC_Comb on the member set than on the non-member set, which shows our synthetic texts are effective for detecting membership signals.

Weakness 2 & Question 2: Clarify the unique aspects of your dual-classifier t-test framework compared to the method in “LLM Dataset Inference”

In Section 3, we show that the “LLM Dataset Inference” fails since it uses the simple null hypothesis that “the suspect dataset was not used for training” with a t-test that compares the MIA scores between the suspect and the held-out sets.

By introducing statistical tests based on dual-classifier, we can statistically distinguish distributional shifts caused by actual membership signals from those caused by generation, and provide more robust detection results. Specifically, our method compares the shifts between suspect and held-out sets with and without MIA metrics, which are measured by the text classifier and the combined classifier, respectively.

Weakness 3 & Question 3: Sensitivity analysis on hyperparameter settings

In our proposed framework, the arbitrator, who verifies the training data usage, can choose hyperparameters in both the held-out data generation and post-hoc calibration stages.

In the held-out data generation stage, the arbitrator can choose hyperparameters according to the AUC of the text classifier, where a low AUC_text indicates nearly IID suspect and synthetic held-out texts.

In the post-hoc calibration stage, there are a few important hyperparameters for our method, especially epochs for training the linear classifiers and t-test sample sizes. We present the sensitivity analysis of our method for these hyperparameters below (we use 200 epochs and 2000 sample size in our manuscript).

The results show that our method can tolerate a wide range of hyperparameters and gives true positives and true negatives.