Fine-tuning can Help Detect Pretraining Data from Large Language Models

Thanks for your recognition and the valuable suggestions. Please find our response below.

1. The setting is realistic in practical [W1, W4]

Thank you for raising the concerns. In this work, we assume the access to fine-tuning the pre-trained models and a few non-member data. We provide a detailed explanation of the two realistic assumptions below.

• Fine-tuning the models: Our method assumes the access to fine-tune the pre-trained models. For the open-source models, we have the ability to fine-tune the models directly. In addition, many API-based commercial models provide the fine-tuning access via API interfaces, enabling task-specific customization, such as GPT-4o [1]. Therefore, our method can be broadly adopted for all open-source models and many popular commercial APIs, which is sufficient to support the development of such fine-tuning methods.

• Collecting non-member data: Due to the vast availability of public text data, it is practical to collect non-member datasets in a special domain, by comparing the LLM release date with the data creation timestamp. The process for collecting data is described in Subsection 3.2. Our method can achieve significant results with only 100 unseen data, which makes it practical for real applications. When the non-member data is unavailable, an alternative solution is to fine-tune with member data in our framework, which can also improve the detection performance (but worse than using non-members).

In summary, our method provides an effective solution to utilize non-member (or member) data to significantly improve the detection when the data is available. This highlights the contribution of our method in many real scenarios.

2. The computational cost of fine-tuning. [W2]

Thank you for raising the concerns. In fact, our method only requires 100 examples to achieve significant performance, and the finetuning can be completed in 109 seconds. Thus, our method is high-efficiency with low computational overhead, making it practical for real applications. Please find the detailed analysis in the General Response.

3. Relations to meta-evaluation [W3]

Thank you for pointing out the potential confusion. We are not familiar with the area of meta-evaluation, so we provide the difference according to your description.

1. Motivation: our method employs fine-tuning to produce score deviations to differentiate members and non-members (members with low deviations). Meta-evaluation seems to use fine-tuning to learn the patterns of data that should be detected.
2. Model usage: we use fine-tuned models to calculate the score deviations as a new score for detecting pretraining data. Meta-evaluation uses fine-tuned models to evaluate the effectiveness of LLM evaluators.
3. Data usage(not sure): In this work, we fine-tune the model on a few collected non-members, which are not utilized to evaluate our method during the test phase. We believe this is different from meta-evaluation, which fine-tunes a model on all data that needs to be detected (non-members). In a word, we evaluate our method on examples that did not occur in the fine-tuning phase.

We believe our method is significantly different from meta-evaluation, although both of them employ fine-tuning. Note that the explanation is based on your description, so the comparison might be not precise. If possible, could you share a related work of meta-evaluation, which fine-tunes a model on all data that needs to be detected?

[1] https://platform.openai.com

Thanks for your recognition and the valuable suggestions. Please find our response below.

1. Definition of 'domain' [W1]

Thank you for pointing out the potential confusion due to insufficient descriptions. In the literature on NLP, the term "domain" typically refers to the distinct areas or fields where language is used [1], such as different topics (i.e., business news, medical knowledge, and legal documents). This is universally used to denote some coherent data related to a topic, style or genre [2]. Researchers usually predetermine the domain of the given dataset in some NLP tasks, such as domain adaptation[3] and domain-specific fine-tuning [9]. In this work, we classify domains based on topics (e.g., event data from Wikipedia and arXiv research papers). In practice, we can use a topic from the same source to denote the specific domain.

Here, we provide a simple notation for reference. Let $D =$ {$D_1, D_2, \dots, D_n$ } denotes the large-scale pretraining dataset, which is comprised of data from multiple distinct domains. Where $D =$ {$X_i$} denotes a specific domain within the pretraining dataset. $X_{i}$ represents the text corpus that are sampled from the underlying distribution of the domain $P_{X_{i}}$.

2. How to collect non-member data [W2]

Thank you for the suggestion. In practice, developers can collect non-member data by comparing the LLM release date and data creation timestamp (as described in Subsection 3.2), following previous works [4]. For example, given a task that determines whether some texts from Wikipedia are used for pretraining by LLaMA, we can collect a few events that occurred after February 2023 from Wikipedia as a non-member dataset for fine-tuning. This is because data created after February 2023 are unseen by LLaMA, as the LLaMA was released in February 2023. Note that our method can achieve significant results with only 100 unseen data, so it is realistic to apply our method in real applications. In addition, our framework can also applied to member data if a few of them are available, as shown in the Discussion.

3. The realistic assumptions of our method [W3]

Yes, our method requires unseen data and model probabilities for each token. We clarify that this method is designed to improve the current likelihood-based methods [4,5,6], which assume access to the output probabilities of LLMs. Despite the assumption, those methods are useful and valuable in practice, because the access is realistic for open-sourced LLMs and many commercial APIs, such as GPT-4o. Notably, our method provides an effective solution to exploit non-member data for significant improvement, when a few data is available. This is a unique contribution to this work, and collecting a few non-member data is also realistic in many scenarios.

4. Details about datasets [Q1]

Thank you for the question. In our work, we conduct experiments on four datasets, including WikiMIA [4], ArXivTection [7], BookTection [7], and BookMIA [4]. For the label of membership, the dataset creators use data occurring before LLM's release date as members, and vice versa [8]. This setting is generally adopted in pretraining data detection of LLMs[4,6,7,10]. In addition, the members and non-members in these datasets are basically balanced, as shown in Appendix C.1. The details of the datasets are presented in Appendix B.

In addition, we conduct experiments on datasets that reduce the temporal difference in Discussion. we implement two strategies to mitigate the temporal shift in the dataset: (1) removing timestamps in the dataset (Deletion) and (2) replacing the year of timestamps with 2023 in the dataset (Replacement). As suggested by reviewers, we also add new results on datasets with IID setups [11], which is introduced in the General Response.

5. Fine-tuning does not happen for each example [Q2]

Thank you for the great question. In our experiments, there is no overlap between the fine-tuning dataset and the evaluation data. Therefore, we do not perform fine-tuning on each sample. Specifically, we only collect a few non-members to build the dataset for fine-tuning and these data are not used for evaluation. In addition, our method can achieve significant results with only 100 collected data, as shown in Figure 4.

6. Analysis of different fine-tuning parameters [Q3]

Thank you for the suggestion. As suggested, we conduct experiments on the WikiMIA dataset with different fine-tuning parameters, including learning rate (e.g. 1e-3, 1e-4, 1e-5), epoch (e.g. 1, 2, 3) and LoRA rank (e.g. 8, 16, 32). The table below presents the AUC score for pretraining data detection with baselines and our method under the LLaMA-7B model. The results show that our method is relatively insensitive to LoRA rank and the number of fine-tuning epochs, while a learning rate of 0.001 performs much better than other choices. We add the sensitivity analysis of hyperparameters in Appendix D.3 of the revised version.

7. Fine-tuning on a mix of domains [Q4]

Thank you for the suggestion. Please find the detailed analysis in the General Response.

8. An alternative in the fine-tuning framework [Q5]

Thanks for the question. In the Discussion, we show that fine-tuning with members can also provide some improvements to some extent. While it cannot achieve comparable performance to fine-tuning with non-members, utilizing a few members can be an effective alternative when the non-members are unavailable.

9. Minor suggestions

Thank you for the suggestions. We have fixed these in the updated version.

[1] Ramponi A, et al. Unsupervised domain adaptation in nlp—a survey. ICCL2020.

[2] Plank B. What to do about non-standard (or non-canonical) language in NLP. KONVENS2016.

[3] Wu H, et al. Domain-Adaptive pretraining methods for dialogue understanding. ACL2021.

[4] Weijia S, et al. Detecting pretraining data from large language models. ICLR2024.

[5] Yonatan O, et al. Proving test set contamination in black-box language models. ICLR2024.

[6] Wentao Y, et al. Data contamination calibration for black-box llms. ACL2024.

[7] Duarte A V, et al. De-cop: Detecting copyrighted content in language models training data. ICML2024.

[8] Gao L, et al. The pile: an 800gb dataset of diverse text for language modeling. arXiv:2101.00027, 2020.

[9] Zheng J, et al. Fine-tuning large language models for domain-specific machine translation. arXiv:2402.15061, 2024.

[10] Zhang W, et al. Pretraining data detection for large language models: a divergence-based calibration method. EMNLP2024.

[11] Maini P, et al. LLM Dataset Inference: did you train on my dataset? arXiv:2406.06443, 2024.

Thanks for your recognition and the valuable suggestions. Please find our response below.

1. Results on IID setups [W1]

Thank you for the suggestion. Reviewer oZEC also mentioned concerns about the temporal shifts and suggest to conduct experiments on data from the Pile with IID setups. As reviewers suggested, we present new results on three datasets that are built from the Pile. The results are shown in the General Response.

2. Clarification of intuition [W2, Q1]

There might be some misunderstandings. We do not claim that "fine-tuning increases the gap between distributions". We clarify that our method utilizes fine-tuning to compute the FSD score, which reflects a large gap between members and non-members. In particular, the FSD score measures the deviation of perplexity (or other existing scores) between the pre-trained model and the fine-tuned model. This is motivated by the phenomenon shown in Figure 2: finetuning on a few unseen data decreases the perplexity of non-members but almost keeps those of members unchanged. The intuition is that members already have a low perplexity, which cannot be easily decreased via finetuning. Thus, we utilize the difference in score deviation to provide an effective metric for detection.

Notably, finetuning does not increase the gap between members and non-members (even closer, as shown in Figure 2). Instead, our method measures the score deviation of the same examples after fine-tuning.

3. The computational cost of our method [W3]

Thank you for the great suggestion. We agree that finding a less expensive method is an interesting direction in the future. However, it does not affect the contribution of this work: revealing the phenomenon and proposing an effective method accordingly. In fact, our method only requires 100 examples to achieve significant performance, and the finetuning can be completed in 109 seconds. Thus, our method is high-efficiency with low computational overhead, making it practical for real applications. Please find the detailed analysis in the General Response.

4. Our method supports various fine-tuning techniques [Q2]

Thank you for the question. Indeed, we investigate the effect of different fine-tuning methods in our paper, which is discussed in Section 5. We conduct experiments with 3 fine-tuning methods to fine-tune LLaMA-7B, respectively. The results show that our method can be implemented with different fine-tuning methods and does not rely on LoRA.

5. Finetuning using non-members from a loosely related domain [Q3]

Thank you for the suggestion. Please find the detailed analysis in the General Response.

Thanks for your recognition and the valuable suggestions. Please find our response below.

1. Results of IID setup [W1, Q1]

Thank you for the suggesion. As suggested, we conduct experiments on data from the Pile dataset, with an IID setup [1]. We present the results in the General Response and add it in Appendix D.3.

2. Ablation study with an IID setup [Q2]

Thank you for the great question. To evaluate our method on an IID setup, we conduct experiments on the BookC2 subset of the Pile dataset under the Pythia-6.9B model. Specifically, we randomly sample varying amounts of non-members (0, 50, 100, 150, 200, 250, 300, 350, 400, 450, 500) from the validation set of the BookC2 as fine-tuning datasets. In addition, we sample 1400 members and 1400 non-members from the train and validation sets of the BookC2 to construct a balanced test set of 2800 examples. Importantly, the fine-tuning data are strictly excluded from the evaluation in the testing phase, ensuring unbiased results.

In Appendix D.3, we present the results of fine-tuning on auxiliary datasets with varying sizes. The results show that our method achieves better performance as the size of the fine-tuning dataset increases. Notably, our method is highly data-efficient, achieving significant improvements with only a few non-members for fine-tuning. For instance, our method improves the AUC score of the Zlib method from 0.48 to 0.78, by leveraging only 100 non-member data for fine-tuning.

[1] Maini P, et al. LLM Dataset Inference: did you train on my dataset? arXiv:2406.06443, 2024.

Hi Reviewer oZEC,

Thank you for the reply and raising the score. Now all reviewers recognize the significance and contribution of this work. In the experiments of the current version, we use 4 datasets with risk of temporal shift and 20 datasets with the IID setup, which is extensive and solid.

For the concern of presentation, we will definitely place the average results of 20 datasets from Pile in the main paper of the final version, and emphasize the importance of these datasets. This is not updated in the discussion, as we were waiting for the feedback of the reviewer, in case of any other suggestions on experiments. We will accomplish this in the final version, because the deadline of uploading a revised PDF was due before the reviewer's response.

Thank you again for your efforts on improving this work.

Best wishes,

Authors