Blackbox Model Provenance via Palimpsestic Membership Inference

We thank the reviewer for their time and effort. We address their questions and concerns below.

Weakness 1 (Definition 2)

Definition 2 aims to establish our primary assumption that the training algorithm $A$ shuffles the data order, which we use for the following algorithms. We will clarify this definition in the revision.

Weakness 2 (Membership inference context)

Thanks for the suggestion! We acknowledge that in the literature, “membership inference” refers to predicting whether an individual example is in the training dataset. Here, we are extending the definition following [1, 2]: instead of focusing on the membership of an individual example, or the membership of a set of examples (e.g., dataset inference), we consider the “palimpsestic membership” as whether "a sequence of examples" is a subsequence of the training examples. We agree that this is not the standard implication of membership inference. We will add more explanation for this in the revision.

Our work is also related to membership inference attacks. The features we use are commonly used in membership inference attacks, e.g., logprobs and logprobs with a reference. We believe better features discovered by the membership inference community could also help improve our test.

Weakness 3 (Typo)

We apologize for the typo; single_kgrams should be matches.

Weakness 4 (Exact training order in real-world settings)

We agree that reconstructing and releasing the full transcript might not be feasible in practice. However, we would like to clarify that our method only requires knowing the relative order of a small sample (<0.0001%) of the training data, and Alice does not need to release the full copy of her training data publicly. Specifically, for technical constraints such as data streaming and multi-phase loading, as long as Alice can track the relative order of a small subset of data, for example, logging a few thousands training examples once in a while, she can run this test. For privacy constraints, disclosing a full copy of training data is only required for public verification under the query setting, in which case Alice can choose to release a subset from a commonly used, CC licensed corpus, such as Wikitext (also see our response to Reviewer zdZQ, weakness 1).

Weakness 5 (Sampling strategies for the sample setting)

Please refer to our response to question 2.

Question 1 (Obtaining the training transcript for pretrained models)

For Pythia and OLMo, the training transcript is reconstructed using a global index, i.e., a mapping from training example order to the order in the original memory-mapped dataset. This global index is either directly provided by the model developers, or can be deterministically regenerated from the random seed and the total number of examples through a shuffle, ensuring the exact training order is reproducible.

Question 2 (Varying Bob's sampling strategies (temperature and top_p) for the sample setting)

We present additional experiments varying decoding strategies (temperature and nucleus sampling) and the prompt distribution in the sample setting and present the p-values. These are using a different method from the paper; here, we count the number of n-gram matches from each training step, and take the Spearman correlation between the number of matches and the training step. (In Algorithm 2, this would be taking a frequency count (like collections.Counter) over matches, and taking the Spearman rank with arange(100000)). Using the Spearman correlation this way is valid for the same reason it is valid for the query setting.

We vary the temperature for Pile completions for the pythia-6.9b-deduped model, compared with itself, and keep top_p=1.0. With lower temperatures, we find the p-values are larger, but relatively low. At 50000 texts (each 128 tokens), the results are relatively stable across temperatures. We then vary top_p and find similar results. Sometimes the p-values can be noisy, so in the revision we will report distributions of p-values for different samples of $N$ texts.

p-values, varying temperature:

p-values, varying top_p:

We also try varying the prompt distribution by sampling prompts from Wikitext (also for the pythia-6.9b-deduped model). We report the p-values below and find they are also significant at 50000 texts.

(For a sanity check, these are the p-values for using completions from the pythia-6.9b model compared with the pythia-6.9b-deduped training order, which we see are not significant.)

We appreciate the reviewer's comments and are happy to discuss further!

[1] Pratyush Maini, Mohammad Yaghini, and Nicolas Papernot. Dataset inference: Ownership resolution in machine learning. In International Conference on Learning Representations, 2021.

[2] Pratyush Maini, Hengrui Jia, Nicolas Papernot, Adam Dziedzic. LLM Dataset Inference Did you train on my dataset? In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.

We thank the reviewer for their time and effort. We address their questions and concerns below.

Weakness 1 (Discussion of model provenance)

This is an excellent point — over time, it is possible that model A could influence model B (or human text that it is trained on) indirectly, and we agree that this is relevant discussion for "model provenance". The main issue [1] raises is a lack of a clear null hypothesis with respect to training data composition. Crucially, our methods are valid regardless of dataset composition, and we have a clear null hypothesis (independence with A’s training order).

We are careful not to claim that our methods can show Bob has necessarily fine-tuned Alice’s model. It is true that if our method labels a model and set of texts as independent, it’s still possible they share some other provenance (not “truly independent”). In the future, one can imagine model providers may run our tests on large samples of human text in order to re-establish a (approximate) null baseline. We will include this discussion in the revision.

Question 1 (Multiple epochs of training)

Unfortunately we could not test this for large models trained for multiple epochs as none are fully open-source (OLMo and Pythia models are trained for up to 1.5 epochs). We show that at least after an additional 0.5 epochs, we can still use the first-epoch order to get significant p-values (Table 2 in the appendix), and there are strong correlations with the last four epochs’ orders for TinyStories models (Appendix A.4). In practice, one could use the training order of the final epoch for the best signal and we believe our method would still work in that setting.

We appreciate the reviewer's comments and are happy to discuss further!

We thank the reviewer for their time and effort. We address their questions and concerns below.

Question 1 (Transparency)

This is a good question. Transparency is important so that the test can be used by third-parties and auditors, for example. It is also important if a model developer (Alice) wants to prove that Bob’s model is a derivative —- she must disclose the secret test set or fingerprint so another party can independently verify the result.

Question 2 (Pythia training data)

In general, Pythia and OLMo construct the training data by first filtering (include deduplication) and then apply a global shuffle. As a result, the training data of pythia-deduped is shuffled independently of pythia. The implementation details can be found in the pythia and OLMo git repositories.

We use the training scripts provided by EleutherAI, in which they provide the seeds. Pythia and OLMo generate the training data by filtering / deduplicating then shuffling, so the pythia-deduped and pythia datasets are shuffled independently.

Question 3 (Figure 1c clarification)

Yes, we mean the differences in losses (e.g. subtracting the loss of a reference model), although the order can also be seen in 1(a), which is just the loss. These plots are mostly to demonstrate that the model in fact preserves the training order, which is more visible when using a reference model. We will add more explanation in the revision.

Question 4 (Fine-tuned models for the sample setting)

Yes, our current experiments in the paper are for the same model for Alice and Bob; we provide additional experiments for the fine-tuned setting in the discussion for Weakness 1.

Question 5 (Are OLMo-0424 and OLMo-0724 both used?)

Yes, in Figure 2 (b) in the main text, Table 3, and Table 4 in the appendix, we show results for fine-tunes of both OLMo-0424 and OLMo-0724.

Weakness 1 (Additional experiments in the sample setting for fine-tuned models)

We provide some additional experiments for fine-tuned models, specifically for later checkpoints of the pythia-6.9b-deduped model. These are using slightly a different method from the paper, which we found it’s more effective; here, we count the number of n-gram matches from each training step, and take the Spearman correlation between the number of matches and the training step. (In Algorithm 2, this would be taking a frequency count (like collections.Counter) over “matches”, and taking the Spearman rank with arange(100000)). Using the Spearman correlation this way is valid for the same reason it is valid for the query setting. We use the base model pythia-6.9b-deduped at checkpoint 100000 (Alice) and generate $N$ texts (Bob’s texts) from checkpoints 105000, 110000, 115000, 120000, and 130000.

We find that we get low p-values even after fine-tuning for 30% the number of pretraining tokens, if we use enough generated texts.

Weakness 2 (API logprobs)

This is true, but logprobs can be estimated from an API with repeated sampling (we show this is feasible under a reasonable budget in Appendix A.6 and C.2), or one could use the sample setting.

We also thank the reviewer for their additional suggestions and will implement them in the revision. We appreciate the reviewer's engagement and are happy to discuss further!

We thank the reviewer for their time and effort. We address their questions and concerns below.

Weakness 1 (Disclosure of Alice's training data)

We understand the reviewer's concerns on disclosing information about the training data. We would like to further clarify when and what Alice needs to disclose about her training data.

Publishing information of training data is only needed for third parties to verify the test result. The main use case of this test is for the model developer (Alice), who clearly has her full training data, to check whether someone else has used their model. In this case, Alice can keep her training data private.

If she wishes to prove to someone else that others are using her model (for public verifiability purposes), she can publish the order of a dataset without privacy concerns like Wikitext. Specifically, for public verification,

• under the query setting, Alice needs to disclose usually less than 1M sequences of her training data. As we discuss on L323-324, Alice can choose which subset of the training data to make public. For example, she may choose to release Wikitext samples. Wikitext is used in training nearly all language models and is under the Creative Commons license, which would not be a privacy concern.

• under the sample setting, Alice does not need to disclose any exact copy of her training data. She only needs to release aggregated n-gram stats of the data for some small n. Given the nature of information that needs to be disclosed, we believe the privacy risk can be mitigated (especially if the subset is something like Common Crawl where releasing it would not have privacy concerns).

Prior methods like [1] have not considered this public verifiability perspective, which is why they do not need to publish test implementation details. Also, we would like to emphasize that inference-time watermarking methods like [1] are not applicable to the sample setting, as Bob can directly sample text himself without the watermark (L112-L114).

Weakness 2 (Retraining on the shuffled data)

This is indeed an interesting question! We have studied this adversarial setting in Section 4.3.3. In our experiments in TinyStories models (Appendix A.4), we find retraining on the same texts with a different shuffle still preserves the orders of the recent iterations. Specifically, we tried retraining for 10 epochs on TinyStories, and the orderings from the last 4 epochs all yielded significant correlation with the final model (Figure 6 of the appendix). We agree that with enough epoching over the same texts, the original order could be lost, however, this means to fully mask Alice’s original training order, Bob would have to do fine-tuning at the scale of the original pretraining budget. In fact, on Pythia and OLMo models, we find that even after retraining 50% of the original pretraining texts (L289-L291 and Figure 5), our tests still yield extremely low p-values. Moreover, fine-tuning on a small subset of the transcript, such as one million sequences, is not sufficient to evade the test. In that case, Alice can use another random subset of her training data, which very likely Bob has not retrained on.

Regarding memorization, we would like to clarify that the most interesting finding is that, in the multi-epoch setting, earlier pretraining orders are inscribed into the model along with later ones (L292-L293), instead of being completely overwritten. This is exactly why we call it “palimpsestic” memorization.

Weakness 3 (Catastrophic forgetting in smaller models)

As the reviewer noted, we show that all failure cases have much higher loss on general language modeling, which means the fine-tuned model’s quality is severely affected. Doing such fine-tuning defeats the purpose of modeling stealing, as these fine-tuned models would not be very usable in practice.

Moreover, larger models are the most likely target of model stealing and misuse, and our test yields low p-values on all 7B and 12B models tested. This is in line with existing studies that show smaller models memorize less [2, 3].

[2] Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2023. Quantifying memorization across neural language models. In The Eleventh International Conference on Learning Representations.

[3] Pietro Lesci, Clara Meister, Thomas Hofmann, Andreas Vlachos, and Tiago Pimentel. 2024. Causal estimation of memorisation profiles.

Question (Sampling training data from the first epoch in experiments)

We sample from the first epoch for consistency across model families (for example, pythia models are trained for one epoch, whereas pythia-deduped models are trained for more than one epoch). The fact that our tests yield such results even using only the earliest epoch ordering shows how effective it is against continued pre-training or fine-tuning.

Limitation (Prior work and disclosure of Alice's training data)

To run this test, Alice would have to disclose a subset of her training data (discussed in weakness 1). However, this subset could be something like Wikitext that would not be proprietary. The most-relevant prior work is [4] which requires using a held-out validation set that would also have to be disclosed to run the test. In particular, this validation set has to be i.i.d., i.e., must be representative of the entire training data, and would in fact be more proprietary.

[4] Pratyush Maini, Mohammad Yaghini, and Nicolas Papernot. Dataset inference: Ownership resolution in machine learning. In International Conference on Learning Representations, 2021.

We appreciate the reviewer's comments and are happy to discuss further!

Thank you for the authors’ detailed responses. After carefully reading through the paper and other reviewers’ comments, I still have several concerns that I hope the authors can address further:

1. Practicality of Sampling Requirements The proposed method requires a large amount of text (often exceeding $10^5$ tokens) for effective detection. In realistic scenarios, obtaining such a volume of pure samples originating solely from a single LLM is often impractical.

   • For example, in many real-world cases, an adversary (Bob) may generate content using multiple LLMs, mixing outputs from Alice’s LLM with other LLMs. In such mixed-generation settings, how does the method ensure that the statistical correlation attributed to Alice’s training order is not diluted or masked by contributions from other LLMs?
   • Additionally, in the sample setting, Alice cannot guarantee that the collected text spans are all generated exclusively by her LLM. A common scenario could involve Bob using Alice's LLM as part of a novel-writing pipeline that includes outputs from various LLMs. And the Bob can adopt paraphasing attack on the generated texts. How would the proposed method handle common scenario?
   • Similarly, in query-based settings, Bob could employ a mixture-of-experts (MoE) setup, dispatching different queries to different LLMs (including Alice’s). How robust is the proposed independence test in these adversarial hybrid-LLM configurations?
   • Furthermore, how would the approach handle situations where Bob blends Alice’s LLM with independently trained LLMs (e.g., via LLM merging or weight averaging)? Does the detection power degrade gracefully in these mixture scenarios, and is there a lower bound on the contribution from Alice’s LLM required for successful detection?

2. Comparison with Learning-based Watermarking Methods Recent works [1][2][3] have proposed learning-based watermarking methods that embed watermark signals directly into LLMs during training or fine-tuning, enabling downstream detection through LLM outputs. These methods generally require significantly fewer tokens for detection compared to the proposal in this paper.

I found your responses to other reviews intersting. It is clear that several aspects of the paper need to be corrected, revised, expanded with additional details, or even added where content is currently missing. Could you please provide a comprehensive list of the changes you are committing to make in the paper and appendices?

[1] On the Learnability of Watermarks for Language Models.

[2] Watermarking Makes Language Models Radioactive.

[3] Towards Watermarking of Open-Source LLMs.