A Statistical Approach for Controlled Training Data Detection

First, while I appreciate the authors’ effort in trying to provide some theoretical guarantee on the result, it is apparent to me after reading that multiple steps of the approach remain ad hoc and thus the guarantee actually means little. For instance, the choice of using “the L2 norm of the model’s gradient as the score in KTD”, which is rather important in your method, must be properly justified. That is, why is it a meaningful choice and how do users understand the guarantee based such as scores then? Furthermore, how do you determine the number of samples? Can you provide some statistical analysis for determining the number of samples (e.g., in the spirit of hypothesis testing for instance)?

Secondly, the experimental evaluation can be improved. For instance, can you conduct some experiments to show the effect of different KTD scores? (Intuitively, one may consider choices such as influence - a near zero influence of a certain sample would imply it is not in the training set). Furthermore, can you conduct the experiments on even larger models such as Llama3?

The following are a list of detailed comments.

Page 1: “Detecting training data for large language models (LLMs) is attracting growing attention, particularly in applications that require high reliability. While numerous endeavors have been made to tackle this issue, ensuring controlled results often remains overlooked.”

Comment: I have trouble inferring what is the problem to be solved from this sentence.

Page 1: “To address this issue, recent studies have focused on detecting training data from large language models (LLMs) …”

Comment: The short-hand “LLMs” have already been introduced.

Page 2: “While the knockoff statistic from the vanilla KI method can be directly applied to KTD, it has a significant drawback …”

Comment: You should introduce what the vanilla KI method is.

Page 3: “The knockoff framework was first proposed in Barber & Candes (2015) as a data-driven method to control the FDR in variable selection for sparse regression problems.”

Comment: As an outsider, I would really appreciate if the authors say at this point how knockoff works - without a proper understanding on how it works, much of the followup discussion doesn’t make sense.

Page 3: “For two number a and b, let a ∨ b represent min(a, b).”

Comment: I find this notation disturbing. Why?

Page 7: “Specifically, we fine-tune the models using the training samples while ensuring that the non-training samples remain unseen by the models.”

Comment: It is not clear to me why it is necessary to fine-tune the models using the training samples - as they are used in the training already I suppose? Also, such fine-tuning might make the model unnecessarily overfitted on these samples, which makes detecting them easier than what should be the case.

• FDR control bounds from Proposition 2 do not seem to hold in general (see Figure 1), and in particular is violated for the important regime of low FDR
• Complete Assumptions for asymptotic optimality (Theorem 1 and 2) are not clearly stated. In particular, Lemma 1 seems to be more of an Assumption than a Lemma (which does not necessarily hold in practice).
• Important experimental setting details are missing, e.g., exact fine-tuning setup.
• Important ablations over the concentration of relevant samples in the training set and the ratio of training to non-training samples in the detection set are missing.
• Related work from membership inference with a similar focus on FDR (Carlini et al., Mireshghallah et al.) and contamination detection (Dekoninck et al. A) is not mentioned or compared.
• Limitations of the work are not discussed, e.g. the need to compute several gradients wrt model weights and thus applicability only to open weight models and the inability to detect training on rephrased data (Dekoninck et al. B).)

References

• Carlini et al. "Membership inference attacks from first principles."
• Dekoninck et al. A "ConStat: Performance-Based Contamination Detection in Large Language Models."
• Dekoninck et al. B"Evading data contamination detection for language models is (too) easy."
• Mireshghallah et al. "Quantifying privacy risks of masked language models using membership inference attacks."

1. Computational Efficiency: The method requires multiple rounds of rephrasing and inference, resulting in higher computational costs compared to baseline approaches. While these costs are reasonable given the improved performance, the paper would benefit from explicitly discussing this computational trade-off.

2. Performance Limitations: For small values of q, the False Discovery Rate (FDR) control becomes less strict, which impacts the method's effectiveness. But I think this limitation is acceptable within the broader context of the method's benefits.

3. Novelty Assessment: The core method is similar to the foundation established by [1,2]. While the paper's primary contribution lies in developing a theoretical framework around this existing idea, the incremental nature of this advancement should be considered when evaluating its impact.

[1] Mattern, Justus, et al. "Membership inference attacks against language models via neighbourhood comparison." arXiv preprint arXiv:2305.18462 (2023). [2] Fu, Wenjie, et al. "A Probabilistic Fluctuation based Membership Inference Attack for Generative Models." arXiv preprint arXiv:2308.12143 (2023).

• The conceptual contribution is marginal: the KTD method is a somewhat direct adaptation of KI with more than one sample.
• It is unclear what the value of Def 1 is: the requirements cannot be guaranteed on sampling, and it is unclear where it is used in the proofs (I checked the proofs, but perhaps it is in some of the works this paper is citing).
• Not major, but there is something unclear in the problem definition, namely what is D_train. Initially, in Background, X1...Xn are samples to be tested, partitioned into D_train and else. These samples may only be a small part of the training dataset, as it happens in evaluation say. Yet later in 4 (line 181), it defines \theta = Alg(D_train), so D_train is the entire training dataset. Or may be it means that the model is trained on some data and further fine-tuned on D_train. This should be clarified.
• In equation 4, t_j / t is a function used to compute the Ws, yet in equation 7 t is again used, but now it is not a function but its bounding Ws. This should be clarified.
• There is something questionable in the problem formulation: it gives FDR over a set of samples which is determined per equation 7 from a joint bound \tau. Why not solve the problem per single sample one-at-a-time, why does it have to be joint?
• Also given the threshold \tau is jointly computed, it seems possible for one (or two, etc.) difficult (call it adversarial) samples to affect the other (easy) samples which may otherwise require much fewer samples, in some sense requiring much larger overall 'm'. In a sense, the method is not adaptive. This doesn't seem to happen in the evaluation, but I can see a situation where it is possible that one trains/fine-tunes with specially designed-paraphrased adversarial samples which trigger this behavior.
• Experimentally, it seems that the larger the model is, the less one needs the generalization proposed by the paper and the higher values of m as the model memorizes more. For Pythia and GPT-neo (around 1B models), even very few samples seem enough. How would the graph look like with standard 7B models (llama, etc.)? Would m=1 essentially work meaning KI is enough? Can one evaluate a 7B model, there are plenty of those around or perhaps a gemma-2.6b model.