STAMP Your Content: Proving Dataset Membership via Watermarked Rephrasings

We thank the reviewer for their positive assessment of our work. We respond to the raised concerns below.

Re: Experimental Designs

E1: Main limitation is that the emphasis is mostly on protecting benchmarks, while the method makes more sense for other types of texts

We would like to clarify that our study is not limited to protecting benchmarks. Our focus is on the broader problem of dataset inference. We conduct preliminary experiments on detecting benchmark contamination due to its importance and the strict requirements for ensuring quality of the rephrased versions. Importantly, in Section 5, we demonstrate STAMP can successfully detect membership of diverse real-world content including blog articles and research abstracts.

E2: if one is okay to keep some versions of the benchmark private, than it couldnt they keep the original benchmark private?

We agree with the reviewer that private benchmarking presents one solution to the contamination problem. However, as highlighted by previous work [1], private benchmarking raises concerns about transparency. We believe our work contributes to an ongoing discussion on ensuring trustworhty evaluations and provides a method to detect contamination for public benchmarks.

Essential References

R1: Radioactive Watermarks (also Q1)

In [2], authors demonstrate that it's possible to detect if an LLM is fine-tuned on the outputs of another watermarked LLM, by detecting the watermark signal in the outputs. A follow-up work [3] (posted the ICML deadline) extends this framework to detect benchmark contamination through watermarked rephrasing, similar to our approach. We find that their approach has limited applicability since it requires the rephrasing LLM and the contaminated LLM to share the same tokenizer. Moreover, we hypothesise that their approach would require stronger watermarks and higher repetition in training data. We conduct preliminary experiments to verify our hypothesis.

Our results show that while [3] can detect contamination with higher repetitions and stronger watermarks, STAMP significantly outperforms this approach across all settings (lower p-value is better, with p < 0.05 indicating contamination and ~0 denotes vanishingly small p values).

Sampling with a higher temperature

We thank the reviewer for suggesting an interesting experiment (Q2). We use this section to discuss reviewer's concern about whether watermarking is necessary and instead if we could just use a higher temperature.

We would like to note that while one intuition behind our approach to use watermarking is indeed increasing memorisation (due to distortions as pointed by the reviewer), we also highlight another important intuition: using different keys (as hash keys) for different rephrases embeds distinct watermarking signals, reducing token overlap between versions and increasing perplexity divergence specifically in cases of contamination (since the contaminated model would overfit the tokens in the public rephrase).

Our position is that while sampling with a high temperature is important, the use of green/red list watermarking is complementary and helps in improving the sensitivity of our test. We perform some preliminary experiments by sampling with a temperature of $1.2$ and found our initial results to be negative. We compute the std dev. of ppl across rephrasing of each sample and present the 95 percentile value below. Our results show that at higher T ($T$ > 1.2) the model often generates gibberish, which renders our test ineffective to the large and frequent outliers. While we believe in principle, we could calibrate the T better, our initial results show that increasing T beyond a point might not be optimal and watermarking can be complementary to higher temperatures.

Questions:

Q1 and Q2 are addressed above.

Q3: Flaws due to the distortion

This is indeed a valid concern and to address this concern, we performed a human study in Section 5 where we indeed find that 6 out of 24 authors indicated their abstracts could use minor edits, suggesting the rephrases can have flaws. However, majority of the authors found the rephrasing to be acceptable. We believe that this will be less of a concern going forward, as general model capabilities (including paraphrasing) continues to improve. Regardless, we will note this as a limitation in our draft.

---

[1] Bansel et al. Peeking Behind Closed Doors: Risks of LLM Evaluation by Private Data Curators. ICLR 25 blog post
[2] Sander et al. Watermarking Makes Language Models Radioactive. NeurIPS 24
[3] Sander et al. Detecting Benchmark Contamination Through Watermarking. Preprint

We thank the reviewer for their insightful comments and are happy to see that the reviewer enjoyed our writing and found our method novel. We respond to the reviewer’s comments and questions below.

Re: Weaknesses

W1: Scaling to real-world pretrained models.

We acknowledge the reviewer's concern about our detector scaling to real-world models. However as the reviewer points out, our method is better than any other method available today. We believe that with larger LLMs and some amount of repetition (factors known to increase memorization [3]) our method will be effective at the scale of real-world datasets and will, hopefully, show better scaling properties compared to other approach. We would like to refer the reviewer to our response to Reviewer ojw8, where we show that for the same $\frac{token}{parameter}$, larger models yield stronger detection results.

Re: Questions

Q1: I appreciate the ablation done for stamp... Are there any trade-offs to be made to use the watermarked version?

We agree that the version of STAMP without watermarked variants also works well, but we demonstrate that watermarking increases the statistical strength of our test. This suggests better scaling trends with larger pretraining datasets. In principle, using a stronger watermark signal may increase detectability but may hurt the quality and utility of the paraphrased text.

Q2: For the scale of real-world models and benchmarks

(2a): what is the value of private key count you would actually recommend?

We believe our analysis in Figure 2 (right) is largely independent of scale of model and benchmarks. We find that 5-6 keys should suffice, as at this point the empirical average of each test sample closely approximates the true average, a factor we believe is independent of the scale of model and benchmarks.

(2b): And how many data samples would you need for it to be meaningful?

Our test provides statistically significant results with as few as a few hundred test pairs (500-1000). Importantly, as noted in our response to Reviewer aozb, our framework allows a single copyrighted sample to generate multiple test pairs.

(2c): Could you apply any scaling laws to results such as in Figure 3?

Thanks for the suggestion. We conduct additional analyses to apply scaling laws to our results. Using the data points from Figure 3, we fit the power law from [3] (linear relationship between $log$ \text{p value} $$ and $log(D)$). We obtain a good fit (e.g. $r^2$ (goodness of fit) of 0.9 for trivia_qa) with the resulting curves predicting that our method will obtain statistically significant results (p<0.05) upto $\approx10$B tokens.

Re: Essential References:

We agree with the reviewer that while these missing references are not essential, discussing them would help us better position our work.

R1: Watermarking makes language models radioactive

We refer the reviewer to our rebuttal to Reviewer oVrP for detailed discussion on radioactive watermarks including additional experiments. We will include these in the next version of the our draft.

R2: Canary in the big-bench benchmark

We believe that while these canaries were originally designed for a different purpose, in principle, they could serve a similar detection function as Wei et al.'s random sequence insertion method. Though potentially effective in some scenarios, such an approach would face the same limitations we highlight for Wei et al.'s work.

R3: Membership inference attacks cannot prove that a model was trained on your data.

The position paper [3] argues that existing MIAs that use data collected a posteriori for calibration are statistically unsound, a position that aligns with our analysis Zhang et al.'s approach, where we highlight it suffers from a distribution shift. More importantly, we believe our framework meets the criteria of a sound training data proof argued in the position paper. Our test rejects the null hypothesis when the test statistic on the public dataset is unusual compared to private datasets which a priori were equally likely to have been used for training.

C2: Two different citations of Meeus et al. (copyright traps for LLMs).

Thanks for pointing this out! These refer to the same paper and we will fix it in the draft.

---

Once again, we thank you for the constructive feedback. Working on the questions has helped us improve the quality of our analysis. Please let us know if we can address any further concerns.

[3] Kaplan et al. Scaling Laws for Neural Language Models
[4] Zhang et al. Membership inference attacks cannot prove that a model was trained on your data. SaTML 25
[5] Carlini et al. Quantifying Memorization Across Neural Language Models. ICLR 23

We thank the reviewer for their feedback. We respond to the reviewer’s comments and questions below.

Re: Weaknesses

W1: If the goal is to detect copyrighted samples, it seems like a strong assumption to presume that the "defender" has a relatively large dataset of copyrighted samples. It could be the case that they only have a few samples, which might render the method ineffective.

We would like to clarify a potential misunderstanding about the sample complexity of our detection method. We find that our test yields statistically significant results with as few as 400 pairs (Figure 3). Importantly, a single copyrighted sample can generate multiple test pairs. For instance, in our blog post case study (section 5.2), we perform dataset inference using 44 posts by performing dataset inference on a collection of paragraphs from the blog, each paragraph forming a separate test pair. This approach allows our method to work effectively even with limited copyrighted samples, giving content creators flexibility in defining what constitutes a pair for the statistical test.

W2: On the other hand, if the goal is to detect benchmarks in the model's pretraining, I don't think it's realistic to assume that someone could know in advance that a benchmark might be used for a model, then ensure it is paraphrased everywhere it appears on the internet, so they can rely on dataset membership detection later.

We would like to clarify that our approach is not meant to protect existing benchmarks but we offer a solution for future dataset releases. As per our approach, the onus is on the benchmark creators to watermarks their own benchmarks before releasing them online. The creator generates multiple paraphrases, each with a unique watermarked key. One version is released publicly for model evaluation, while others remain private, as explained in section 4.1 of our draft. To detect contamination, benchmark creators can apply our statistical test on any target model to obtain evidence if the target model's training data included their benchmark. We highlight this requirement in Section 7 as a limitation, noting that data must be carefully prepared before public release to enable detection. This constraint is not unique to our approach but is shared by existing works [1,2] and we believe is fundamental for a sound statistical test [3].

W3: Additionally, if all the benchmarks from the model's pretraining were "paraphrased" this could negatively affect performance, as the model might start learning the "watermarks" introduced by the paraphrases.

We would like to point out that an important feature of our work is that it allows benchmark creators to use different private hash keys to watermark the public version of each document in their collection. Given that each dataset only constitutes a small fraction of overall training corpora, and with prior work [4] demonstrating that even training on data with just two different keys doesn't result in models learning individual watermarks, it is unlikely that the model would learn the watermark. Hopefully, this assuages your concern about potential watermarking learning.

Additionally, we experiment (in the section 4.2; under false positive analysis) to detect membership of held out samples that are watermarked with the same key as the contaminated samples, and find that our approach (correctly) does not detect them to a part of the training corpus. This experiment provides further evidence that models do not learn the watermarks introduced by the paraphrasers.

Re: Experimental Design

One potential issue is that the authors consider only continual pretraining... results for regular pretraining... might differ significantly.

We acknowledge the reviewer's concern about the potential recency bias in our setup. While previous research [5] has shown that training order has little effect on memorization, the training dynamics of memorization in LLMs remains an active area of research. While such factors can influence the memorization, our contribution is in demonstrating for any given level of memorization, our test provides greater sensitivity compared to any other existing approach and is robust against false positives.

---

Once again, we thank you for the insightful questions. We hope this response has addressed your concerns and would request you to kindly reconsider your overall assessment. Please let us know if we can address any further concerns.

[1] Wei et al. Proving membership in LLM pretraining data via data watermarks. ACL 24
[2] Oren et al. Proving Test Set Contamination in Black-Box Language Models. ICLR 24
[3] Zhang et al. Membership Inference Attacks Cannot Prove that a Model Was Trained On Your Data. SaTML 25
[4] Gu et al. On the Learnability of Watermarks for Language Models. ICLR 24
[5] Biderman et al. Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling. ICML 2023

We thank the reviewer for their insightful comments and feedback. We are happy to see that they appreciate the robust detectability that our method offers, and find the paper easy to follow. We discuss their concerns below:

Re: Important baselines not discussed (ER1)

Thanks for sharing these baselines, we would like to clarify that in our current submission, we already compare our approach against min-k [1] (along with other popular MIAs) in section 4.2 under baselines and Table 7, and find our approach to outperform these baselines.

Based on your suggestions, we conduct new experiments to benchmark additional MIAs [2,3] and share the detection performance (AUROC) under two settings of non-members:same documents where we use different rephrasing of the same test samples and different documents where we use a held out set of test samples. The AUROC scores $\approx 0.5$ show that such baselines are ineffective in determining membership.

Overall our findings corroborate with recent studies [6,7] that highlight the failure of such heuristic based MIAs. Additionally, as highlighted by previus work [8], such heuristic MIAs do not provide a sound statistical proof of membership. Given these additional experiments, we hope that your major concern is addressed.

Re: Other data contamination papers (ER2)

We respectfully disagree with the assertion that our methodology is "quite similar" to the cited works. [5] primarily studies how inclusion of simple variations of test data (including rephrasings) can artificially inflate benchmark performance. Similarly, [4,5] highlight limitations of existing white-box decontamination approaches. In contrast, our approach utilizes watermarked rephrasings as a component of our proposed framework which provides a principled statistical framework for dataset inference. We will update the draft to discuss and constrast our work from these related papers [4,5].

Re: Weakness: “grey-box” access

We acknowledge your concern about our work requiring grey-box access to model probabilities (also discussed as a limitation in Section 7 of our draft). However, majority of prior work on dataset inference assumes grey-box access, similar to our work. In the current landscape where there is a lack of successful black-box MIAs that provide a statistical proof, alternative approaches can be used: probability extraction attacks [9] or through trusted third parties such as legal arbiters. Importantly, as the field develops better black-box metrics, our statistical framework may be adapted to perform paired tests on those metrics instead of loss values.

Re: Other Comments

It might be helpful to test the method on even larger, more highly capable LLMs. The paper suggests it should generalize well, since bigger models memorize more.

We agree that testing on larger LLMs would be valuable. Existing research [10] shows that larger models memorize training data more aggressively. We present preliminary results comparing models of different sizes, maintaining a proportional token-to-parameter ratio:

The results ($log$ \text{p value} $$ , where lower is better and anything below -3 is statistically significant) suggest that for a fixed $\frac{T}{P}$, our method performs better on larger models, supporting our hypothesis about improved detection with model scale.

Dealing with the rephrased or intended contamination?

Thanks for the insightful question! To clarify, our work focuses specifically on verbatim contamination, as our primary goal is to detect membership of a dataset with statistical guarantees. We will update Section 7 to explicitly discuss this point, and clarify the scope of our study.

---

Once again, we thank the reviewer for the constructive feedback. We hope our response addresses their concerns effectively and kindly request you to reconsider your assessment of our work. Please let us know if we can address any other concerns.

[6] Duan et al. Do membership inference attacks work on large language models? COLM 24
[7] Maini et al. LLM Dataset Inference: Did you train on my dataset? NeurIPS 24
[8] Zhang et al. Membership Inference Attacks Cannot Prove that a Model Was Trained On Your Data. SaTML 25
[9] Morris et al. Language Model Inversion. ICLR 24
[10] Carlini et al. Quantifying memorization across neural language models. ICLR 23