How much of my dataset did you use? Quantitative Data Usage Inference in Machine Learning

1. The authors mention two possible improvements in Appendix G. I think this paper would be really strong if the authors could be more concrete on how to apply one of the ideas to their algorithm and show some results.

We sincerely appreciate your efforts in reviewing the improved methods in the Appendix! In response to your question, we have updated the details of the second-order debiasing methods in Appendix H.2 in the revised paper. However, we maintain the simple first-order dataset-level debiasing method in the main paper, as one of our primary focus is to show that the key to solving DUCI problem (via existing MI techniques) lies in the debiasing concept itself, independent of the specific debiasing design. We believe the conciseness and straightforwardness of main methods are essential for effectively conveying this concept. Additionally, as we have demonstrated, this simple debiasing approach can serve as a foundation for various refinements:

First, the unit of debiasing is adjustable: As shown in Footnote 1 and Table 2, this method can be extended to subgroup-level debiasing to address special sampling scenarios---a general challenge faced by all baselines and previous dataset inference techniques.

Second, it can be enhanced to higher-order debiasing (to membership inference methods can leverage higher-order statistics) to capture the correlation between the membership prediction between records: We present an example of second-order debiasing method in the Appendix H.2 for cases where records in the target dataset are uniformly randomly sampled. However, this example method faces an increased algorithmic complexity (from $O(|X|)$ for first-order debiasing to $O(|X|^2)$ for second-order). Developing more efficient higher-order DUCI methods remains an intriguing direction for future work.

We sincerely thank the reviewer for their valuable comments, which have helped improve the clarity of our work. We have addressed all feedback in the revised paper, with the major revisions including:

Improved clarity of Section 2.1

a. Added definition of $\theta(x)_y$ in Line 98

b. Made the definition of reference model, specific number of population data and how they are used, computation details of RMIA clear in Lines 108-115

c. Removed the expression that could potentially cause confusion in Lines 285–286.

Dependence/correlation of records is handled in a confusing manner.

Thank you for pointing out the potential confusion. We have improved the clarity of Lines 489-497 in the revised version to separate these two different "correlation" terms:

1. Correlation in Lines 444–448 or Footnote 1:
   The term $\frac{\text{Corr}_i(\text{TPR}_i - \text{FPR}_i, p_i)}{\text{TPR} - \text{FPR}}$ refers to the "correlation" between the value of ground-truth sampling probability $p_i$ of record $i$ and the value of $\text{TPR}_i - \text{FPR}_i$. Here, we (slightly abusively) use the term "correlation" instead of "covariance" because neither $p_i$ nor $\text{TPR}_i - \text{FPR}_i$ are random variables. Note that, although the value of $p_i$ may have a correlation with $\text{TPR}_i - \text{FPR}_i$ for each record $i$, in the DUCI pipeline, the $p_i$ is always a fixed constant, and each record $i$ is Bernoulli-sampled according to $p_i$. A more detailed explanation of this has been added to Appendix D.

2. Correlation in Lines 490–491:
   This refers to the correlation between the membership probability predictions $\hat{p}_i$ and $\hat{p}_j$ for different records $i$ and $j$ in the dataset. Specifically, "close-to-zero correlation" here means the membership prediction for one record $i$ does not affect the prediction for another record $j$, i.e., $\mathbb{E}[\hat{p}_i \hat{p}_j] = \mathbb{E}[\hat{p}_i]\mathbb{E}[\hat{p}_j]$ for any $i, j \in [|X|]$.

We hope these clarifications resolve the confusion.

3. Why assuming some joint/mean logits follow a normal distribution? Why the MLE with joint logits is presented as an idealized baseline?

The motivation and rationale behind the MLE baseline design are threefold. Below, we explain them and will make then clearer in our revision.

Rationale for using MLE The likelihood ratio test is theoretically proven to be optimal for binary hypothesis testing by Neyman-Pearson lemma [1]. Given that DUCI can naturally be framed as a multi-hypothesis testing problem when the granularity of dataset usage inference is known (we can assume the idealized baseline has access to the granularity of dataset usage, providing it with more information than our method), MLE serves as a natural extension of the pairwise likelihood ratio test to address the DUCI problem.

Rationale for assuming joint/mean logits follow normal distributions
Approximating logits as Gaussian distributions to compute likelihoods is a widely adopted and effective practice in membership inference literature (even in LiRA [2]). This practice is supported by both empirical evidence and theoretical applications, which show that logits often exhibit Gaussian-like behavior across diverse model architectures, data types, and tasks [2, 3, 4]. Therefore, we adopted this practice to compute likelihoods to construct a reasonable and robust baseline that aligns with proven methodologies. To the best of our knowledge, no alternative approaches currently provide apparently better performance or comparable simplicity.

Necessaty of considering MLE with joint logits
While Avg-Logit MLE perform well empirically, averaged statistics inherently introduce information loss. Therefore, it is necessary to include the Joint-Logits MLE as a performance baseline for a lossless scenario. In an idealized setting, where the ground-truth distribution of logits is known, using the joint logits is the most informative choice. However, its empirical sub-optimal performance can be attributed to the curse of dimensionality, i.e., high-dimensional observations (joint logits) typically exhibit a lower signal-to-noise ratio compared to one-dimensional observations (averaged logits).

[1] Neyman, J., & Pearson, E. S. (1933). IX. On the problem of the most efficient tests of statistical hypotheses.

[2] Carlini, N., Chien, S., Nasr, M., Song, S., Terzis, A., & Tramer, F. (2022, May). Membership inference attacks from first principles.

[3] Top-nσ: Not All Logits Are You Need, Chenxia et al, 2024

[4] Lee, J., Xiao, L., Schoenholz, S., Bahri, Y., Novak, R., Sohl-Dickstein, J., & Pennington, J. (2019). Wide neural networks of any depth evolve as linear models under gradient descent.

Thank you for your important and interesting questions. We believe all our discussion will offer valuable insights for future work.

1. My main concern is that this method requires known training set for estimating FPR and TPR. In practice, the train set is usually private (see following reference Zhang at el 2024).

Thanks for raising this intriguing question. We are aware of the challenges in designing a non-member scenario for evaluating training data proof, as discussed by Zhang et al., 2024. Three key issues in the current mainstream design of evaluation that related to our task are as follows:

1. Distribution shifts: Non-member data collected based on the model's training cutoff date can introduce distribution shifts, making the evaluation artificially easier.
2. Member uncertainty: Not all data released before the cutoff are guaranteed to be members since the cutoff date provided by model developers may be inaccurate.
3. Causal relationships: Using hold-out counterfactual data as non-members can introduce causal dependencies, as the model may have been trained on a related, recently released version of the data.

To ensure reliable evaluation in our book copyright infringement case study, we fine-tune a GPT-2 model on a recently collected (by cutoff date) dataset, ensuring that the original training set has no causal relationship with the new data. Instead of directly treating the entire dataset as non-member data, we fine-tune the model on different proportion of data to create members and non-members. This ensures there is no distribution shift between members and non-members that could be exploited. Our focus here is to develop effective methods for the DUCI problem and demonstrate their superiority over baselines under a fairly designed evaluation, which we achieved.

Regarding the concern about evaluation on large production models: we agree that closed-source training pipelines of production models pose problems. However, this issue is independent of method design, and all training data proof evaluations face the same problems. This calls for more open-source (like Pythia [1]) evaluation benchmarks on different model architectures and datasets.

Lastly, regarding the TPR/FPR values used for debiasing in our method: we do not assume access to the target model's training set. Instead, TPR and FPR are estimated on our private dataset (known to the dataset owner) using a reference model. This reference model could be finetuned based on a checkpoint released before the dataset’s creation or even a smaller model with a different architecture and trained on different data [2,3].

[1] Biderman, S., Schoelkopf, H., Anthony, Q. G., Bradley, H., O’Brien, K., Hallahan, E., ... & Van Der Wal, O. (2023, July). Pythia: A suite for analyzing large language models across training and scaling.

[2] Duan, M., Suri, A., Mireshghallah, N., Min, S., Shi, W., Zettlemoyer, L., ... & Hajishirzi, H. (2024). Do membership inference attacks work on large language models?

[3] Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Kather- ine Lee, Adam Roberts, Tom B Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting Training Data from Large Language Models.

2. In Figure 3, the length of confidence interval seems to be large compared to the absolute error. Is this true?

Thanks for pointing out the potential confusion; we will improve the clarity of the confidence interval (CI) explanation in our paper. First, the small value of MAE compared to the length of the 95% CI illustrates that our predictions are highly concentrated around the true value, as if the unbiasedness of $\hat{p}$ is empirically achieved, MAE equals to the standard deviation $\sigma$ of $\hat{p}$. Second, given that the 95% confidence interval is calculated according to Lyapunov CLT ($\hat{p}$ follows an approximately Gaussian distribution), its length is theoretically equal to $(\mu + 2\sigma) - (\mu - 2\sigma) = 4\sigma$, which is approximately four times the $sigma)$ (MAE).

As observed, the maximum length of the 95% confidence interval (CI) is approximately 0.12, while the maximum absolute error using a single reference model is 0.027 in Table 1—roughly one-quarter of the CI length. This shows that the CI length is quite small and effectively demonstrates: (1) the empirical unbiasedness of our estimator and (2) the high concentration of our predictions around the true value.

We thank the reviewer for the positive feedbacks and helpful suggestions for further improvement!

I don't see any major weaknesses. It would be nice to show a comparison between DUCI and SOTA “binary” data usage algorithms for the specific case for when p=1 and p=0 to demonstrate that DUCI still has comparable performance to “binary” data usage algorithms in these specific cases.

Thanks for the suggestion! Below is the formulation of using DUCI for the binary dataset inference problem, with results demonstrating the effectiveness of our methods in binary scenarios. We have also included this comparison in Appendix G.5 of the revised paper.

Hypothesis Testing Formulation for DUCI in Binary Dataset Inference Problems

Prior binary dataset inference literature consider the binary hypothesis test to solve the problem, where the null hypothesis has a general form $H_0: s = s' + \tau$ (i.e., the target model is not trained on the protected dataset). For different methods, $s$ and $s'$ can take the following forms:

1. Dataset Inference [1]: $s$ and $s'$ represent the distances to the decision boundary measured on a private dataset and a population dataset, respectively. This hypothesis assumes that if the model was trained on the private dataset, the distance measured on the private dataset would exceed that on the public dataset.
2. LLM Dataset Inference [2]: $s$ and $s'$ are the weighted aggregations of 52 MIA scores over the private and population datasets. This hypothesis assumes that merged MIA scores would be significantly higher for members than non-members over enough samples.
3. Backdoor Watermarks [3]: $s$ and $s'$ are the confidence score on the target label given backdoored inputs and given clean inputs. This hypothesis assumes that a model trained on a poisoned dataset (if successfully backdoored) will assign higher confidence to the target class when triggered, but not for clean inputs.

Regarding DUCI, a straightforward simplification from DUCI to the dataset inference problem can be made by set $s = \hat{p}$, $s' = 0$, and $\tau$ serves as a hyperparameter, which may vary depending on the data type. Below, we report the performance of our method adapted for solving the binary dataset inference problem.

Table: Comparison of p-values between DUCI and binary dataset usage algorithms for determining whether a dataset $X$ (size 500) has been used. The complete training dataset of the target model has a size of 25,000. For p-values, a smaller value for Dataset Used is better, while a larger value for Dataset Not Used is better.

Consistent with the performance shown in Figure 1, all methods can perfectly solve the binary dataset usage problem when the significance level is set to common thresholds such as 0.05 or 0.01. For backdoor watermark methods, the main challenge lies in the successful injection of backdoor when the dataset is not fully sampled or when the protected dataset's relative size is small. This will significantly impact performance, e.g., poisoning even 30% of X performs poorly when $|X|$ is small. Our method performs exceptionally well, achieving comparable results to backdoor watermarking when the entire dataset is poisoned. In principle, the performance of Dataset Inference should be close to our method; however, the slight drop in performance may be attributed to the choice of signal, as the loss-based score is less distinguishable in distribution than the likelihood ratio-based score.

[1] Maini, P., Yaghini, M., & Papernot, N. (2021). Dataset inference: Ownership resolution in machine learning.

[2] Maini, P., Jia, H., Papernot, N., & Dziedzic, A. (2024). LLM Dataset Inference: Did you train on my dataset?.

[3] Li, Y., Zhu, M., Yang, X., Jiang, Y., Wei, T., & Xia, S. T. (2023). Black-box dataset ownership verification via backdoor watermarking.

Regarding the Concern About the Performance of Our Method in Determining Dataset Usage

As discussed in Lines 35–46 and shown in Figure 1, while methods restricted to binary predictions under an all-or-none dataset usage scenario cannot ensure consistent predictions for partial utilization, a method providing fine-grained estimates can naturally be reduced to solve the binary problem.

To illustrate this, consider the null hypothesis ($H_0: s = s' + \tau$, i.e., the target model is not trained on the protected dataset) used in prior binary dataset inference literature. For different contexts, $s$ and $s'$ can take the following forms:

1. Dataset Inference [1]: $s$ and $s'$ represent the distances to the decision boundary measured on a private dataset and a population dataset, respectively. This hypothesis assumes that if the model was trained on the private dataset, the distance measured on the private dataset would exceed that on the public dataset.
2. LLM Dataset Inference [2]: $s$ and $s'$ are the weighted aggregations of 52 MIA scores over the private and population datasets. This hypothesis assumes that merged MIA scores would be significantly higher for members than non-members over enough samples.
3. Backdoor Watermarks [3]: $s$ and $s'$ are the confidence score on the target label given backdoored inputs and given clean inputs. This hypothesis assumes that a model trained on a poisoned dataset (if successfully backdoored) will assign higher confidence to the target class when triggered, but not for clean inputs.

For DUCI, a straightforward simplification to the dataset inference problem can be made by set $s = \hat{p}$, $s' = 0$, and $\tau$ serves as a threshold, which may vary depending on the data type. Below, we report the performance of our method adapted for solving the binary dataset inference problem.

Table: Comparison of p-values between DUCI and binary dataset usage algorithms for determining whether a dataset $X$ (size 500) has been used. The complete training dataset of the target model has a size of 25,000. For p-values, a smaller value for Dataset Used is better, while a larger value for Dataset Not Used is better.

Consistent with the performance shown in Figure 1, all methods can perfectly solve the binary dataset usage problem when the significance level is set to common thresholds such as 0.05 or 0.01. For backdoor watermark methods, the main challenge lies in the successful injection of backdoor when the dataset is not fully sampled or when the protected dataset's relative size is small. This will significantly impact performance, e.g., poisoning even 30% of X performs poorly when $|X|$ is small. Our method performs exceptionally well, achieving comparable results to backdoor watermarking when the entire dataset is poisoned. In principle, the performance of Dataset Inference should be close to our method; however, the slight drop in performance may be attributed to the choice of signal, as the loss-based score is less distinguishable in distribution than the likelihood ratio-based score. We did not compare with [2] as combining multiple MIAs is orthogonal to our approach. Our method can debias any number of MIAs using the same reference models without retraining, with combination possible after debiasing if needed.

Finally, it is important to note that directly comparing the reported error of DUCI at $p = 0$ and $p = 1$ to that of dataset inference is inherently unfair. DUCI predicts a continuous value, whereas dataset inference is a simple binary classification task.

[1] Maini, P., Yaghini, M., & Papernot, N. (2021). Dataset inference: Ownership resolution in machine learning.

[2] Maini, P., Jia, H., Papernot, N., & Dziedzic, A. (2024). LLM Dataset Inference: Did you train on my dataset?

[3] Li, Y., Zhu, M., Yang, X., Jiang, Y., Wei, T., & Xia, S. T. (2023). Black-box dataset ownership verification via backdoor watermarking.

Thank you for your thoughtful comments and interesting questions! We greatly appreciate their quality and have provided detailed answers to each one below.

1. Analysis on the approximation error in the debiasing process and the improved clarity of the derivation in Footnote 1

Our analysis shows that the simplification in the debiasing process will not introduce errors in many practical sampling scenarios, such as uniform or i.i.d. sampling (which are the common scenario considered in the long line of prior works in binary dataset inference literature listed in the Introduction section). Only in special cases where there is a strong correlation between the probability of sampling $i$-th point $p_i$ and its $TPR_i - FPR_i$, there would be an error. However, this can be effectively mitigated by subgroup debiasing, as shown in Table 2. We next present the detailed analysis (which is also what footnote 1 analyzed), and we have added details of footnote 1 in the Appendix D.

In Equation (7) and (8), we leverage dataset-level $\text{TPR}$ and $\text{FPR}$ (i.e., $\text{TPR} = \frac{1}{|X|} \sum_i \text{TPR}_i$ and $\text{FPR} = \frac{1}{|X|} \sum_i \text{FPR}_i$) to replace the individual $\text{TPR}_i$ and $\text{FPR}_i$. This simplification avoids the computationally cost (or large sampling errors) of debiasing each $\hat{p}_i$. This simplification is justified because the proportion $p$ is a dataset-level statistic, as analyzed below. To avoid confusion, we introduce $\tilde{p}$ and $\tilde{p}_i$ as the estimators under dataset-level debiasing, and we next prove $\tilde{p} = \frac{1}{|X|} \sum_i \tilde{p}_i = \frac{1}{|X|} \sum_i \frac{\hat{m}_i - \text{FPR}}{\text{TPR} - \text{FPR}}$ is an unbiased estimator of $p$ whenever a correlation term between $\text{TPR}_i - \text{FPR}_i$ and $p_i$ is zero: Given

$

\mathbb{E}[\tilde{p}] = \mathbb{E}\left[\frac{1}{|X|} \sum_i \tilde{p}_i\right] = \frac{1}{|X|} \sum_i \mathbb{E}[\tilde{p}_i] = \frac{1}{|X|}\sum_i \mathbb{E}\left[\frac{\hat{m}_i - \text{FPR}}{\text{TPR} - \text{FPR}}\right]

$

Plugging Equation (5) into the above equation, we can get:

$

\mathbb{E}[\tilde{p}] & = \frac{1}{\text{TPR} - \text{FPR}} \cdot \frac{1}{|X|} \sum_i \left(p_i \cdot \text{TPR}_i + (1 - p_i) \cdot \text{FPR}_i - \text{FPR}\right) \\
& = \frac{\frac{1}{|X|} \sum_i \left[p_i \cdot (\text{TPR}_i - \text{FPR}_i)\right]}{\frac{1}{|X|} \sum_i \left[\text{TPR}_i - \text{FPR}_i\right]}

$

Given $p = \frac{1}{|X|}\sum_i p_i$, note that

$

\frac{1}{|X|} \sum_i \left[p_i \cdot (\text{TPR}_i - \text{FPR}_i)\right] = \frac{1}{|X|} \sum_i p_i \cdot \frac{1}{|X|} \sum_i (\text{TPR}_i - \text{FPR}_i) + \text{Corr}_i(\text{TPR}_i - \text{FPR}_i, p_i).

$

(Here, we use the term "correlation" instead of "covariance" because $\text{TPR}_i - \text{FPR}_i$ and $p_i$ are not random variables.) Thus, we have:

$

\mathbb{E}[\tilde{p}] = p + \frac{\text{Corr}_i(\text{TPR}_i - \text{FPR}_i, p_i)}{\text{TPR} - \text{FPR}}.

$

The correlation term suggests that for many practical sampling methods (e.g., uniform sampling, i.i.d. sampling), this simplification results in an unbiased estimator for $p$ because the correlation is 0. For specialized sampling methods, subgroup debiasing can ensure (empirical) unbiasedness, as discussed in Table 2, by making $\text{TPR}_i - \text{FPR}_i$ constant within each subgroup. This ensures that the correlation term for each subgroup is 0, providing a group-level debiasing approach. Note that the term "correlation" (slightly abused here) is used in the context of how the value of $p_i$, a pre-fixed constant in the DUCI pipeline, is determined. This is distinct from the correlation between membership predictions in Figure 4.

Figure 2 can be clearer if the x axis is in log scale; Missing relevant refs: Kandpal et al for membership inference of users (groups of data) and is related to dataset inference, Vyas et al for copyright protection, Zhang et al. for a recent MIA

Thanks for the suggestions. We have updated in revised version.

6. Further, how does the proposed method work if our goal is to provide a multiplicative guarantee of the form that $\hat{p}/p \in (1/c, c)$? These would be more realistic in the small regime.

This is an important and relevant question. However, according to the standard packing argument [1], providing a multiplicative guarantee for DUCI (regardless of the method used) is fundamentally impossible. A straightforward example to think about this is: in the extreme case where the ground truth $p = 0$, $\hat{p}$ must also equal 0 to keep the multiplicative ratio bounded. Below, we provide a more detailed explanation.

Let the ground truth membership probability for record $i$ in the training dataset be $p_i$. By definition, a sampled dataset generated under sampling probabilities $(p_1, \ldots, p_n)$ and $(p_1 \pm \frac{1}{10n}, \ldots, p_n \pm \frac{1}{10n})$ can be identical with at least constant probability $\frac{9}{10}$ based on the union bound. As a result, with constant probability, a DUCI algorithm cannot reliably distinguish between datasets sampled with $(p_1, \ldots, p_n)$ versus $(p_1 \pm \frac{1}{n}, \ldots, p_n \pm \frac{1}{n})$, leading to an unavoidable additive error of $\frac{1}{n}$ on either $(p_1, \ldots, p_n)$ or $(p_1 \pm \frac{1}{n}, \ldots, p_n \pm \frac{1}{n})$. Consequently, for any fixed $c \geq 1$, as $p_i \to 0$, the multiplicative error $\frac{\hat{p}_i}{p_i}$ either grows to infinity or shrinks to zero, falling outside the range of $(1/c, c)$.

However, as discussed in Lines 408–410, additive error is a more consistent and meaningful metric for the DUCI problem. This is because DUCI is fundamentally a discrete counting problem, where the focus is on the number of incorrect counts, making additive error a more appropriate measure. For instance, in a small protected dataset of size 10, the unit of the additive error rate is 0.1, which consistently corresponds to a single misprediction. In contrast, using (relative) multiplicative error will lead to nonsensical results: a single misprediction when $p = 0.1$ produces the same ratio as mispredicting all 10 points when $p = 1.0$, which is clearly unreasonable.

7. Like differential privacy is designed to protect against membership inference, are there any provable protections against DUCI?

Indeed, when the training algorithm satisfies differential privacy, it is possible to prove upper bounds for the error of DUCI (still via standard packing argument [1]). Loosely speaking (as our goal here is not to derive a tight bound for DUCI), given the ground truth membership probability be $p_i$ for record $i$ in the training dataset, the sampled dataset under sampling probability $(p_1, \cdots, p_{n})$ and $(p_1\pm\frac{1}{n\varepsilon}, \cdots, p_n\pm\frac{1}{n\varepsilon})$ only differ by at most $\frac{1}{\varepsilon}$ records in expectation by definition. Thus under $\varepsilon$-DP training algorithm, with constant probability any adversary could not distinguish between the datasets sampled by $(p_1, \cdots, p_n)$ and $(p_1\pm\frac{1}{\varepsilon n}, \cdots, p_n\pm \frac{1}{\varepsilon n})$. This in turn, causes an inevitable MAE of $\frac{1}{\varepsilon n}$ for estimating the dataset usage $\frac{1}{n}\sum_ip_i$ under $\varepsilon$-DP training algorithm. It is an interesting open question regarding the connections between DP auditing via DUCI versus prototypical DP auditing experiment (e.g., via repeated retraining runs).

[1] Hardt, M., & Talwar, K. (2010, June). On the geometry of differential privacy.

8. Why do you think MIA Guess fails to work?

The high-level reason, as discussed in Section 3.2 Errors in Optimal Point-Wise Membership Inference, is that per-point MIA can make errors. These errors may arise from the intrinsic randomness of the algorithm or inability to capture precise membership information from model outputs. Under DUCI, these errors accumulate across the training set, causing the aggregated MIA guess to deviate significantly from the true $p$. The specific reasons of these errors are not fixed as discussed in prior works [2,3].

Some sources of error are intertwined. For instance, when the score used for MIA is challenging to be perfectly normalized across data points, the optimal threshold may vary for different points. In such cases, naive threshold sweeping becomes less effective. Methods like RMIA, which apply thresholding to the rank of score within the population distribution rather than directly thresholding the score, can be more robust in these scenarios.

However, the objective of this work is not to design the best MIA but to debias any given MIA to perform effectively in DUCI. As such, we do not delve into the specific limitations of existing MIAs.

[2] Aubinais, E., Gassiat, E., & Piantanida, P. (2023). Fundamental Limits of Membership Inference Attacks on Machine Learning Models.

[3] Maini, P., et al. (2024). LLM Dataset Inference: Did you train on my dataset?