Empirical Privacy Variance

We thank the reviewer for initiating the interesting discussions.

Let us begin by explaining 1) what we mean by “empirical privacy”, 2) our empirical privacy measures, and 3) the choice of computing the average score.

1. As noted in the introduction, we take a practical perspective on empirical privacy through users’ perceptions of model behaviors. This is motivated by the gap between the increasing use of DP in LLMs, and the tangible privacy risks that arise in user interactions. We believe empirical privacy should hold richer meanings in this context, extending beyond what is usually referred to (privacy attacks).

2. Our considered empirical privacy measures largely follow this standpoint. They are based on memorization and are specific to the generative nature of LLMs, rather than being generic such as the AUC of membership inference which can be applied to any ML model.

3. Finally, we average the empirical privacy scores over a set of secrets because i) this is how these metrics are evaluated in the literature (e.g., the works that proposed ACR and VMR); ii) we aim to ensure that the conclusions we draw are robust.

Circling back to the reviewer’s comments,

Average- vs. worst-case. We thank the reviewer for the insightful comment. We agree that this adds another layer of subtlety which we overlooked. We thus re-examined whether switching from average- to worst-case measures would affect our conclusions. It turns out that they remain unchanged: empirical privacy variance persists; the regression study yields similar qualitative results; the correlation between empirical privacy scores and audited values remains low.

These results suggest what’s critical is not the distinction between average- and worst-case empirical privacy measures. Instead, the fundamental gap lies between what DP promises (preventing reidentification/membership inference) vs. how we measure privacy (whether a model generates a specific secret given a prompt). This gap is different from the one raised by the reviewer (per-instance guarantees vs. worst-case guarantees).

Furthermore, we believe it is not appropriate to view our empirical privacy measures as representing the "average of the per-instance guarantees" or “expected leakage over datapoints” as: 1) we focus on a small subset of secrets that exhibit strong sign of memorization (Appendix E.5); and 2) a secret is only a small substring of an instance.

We will add the above discussions and the new results in the camera-ready.

Privacy auditing. While we agree with the reviewer that the “nice controlled study” would be great to have, there are technical challenges that prevent this. 1) Empirical privacy measures on datapoints used for auditing: Auditing uses specially crafted canary samples; they are long and each member is inserted once in the training data. Applying empirical privacy measures on them will likely yield insignificant results. 2) Auditing on extracted secrets used for empirical privacy measures: The secrets are pieces within training samples but not whole samples; they could occur in more than one sample. These create mismatches with the requirement on data in auditing. Moreover, the size of the secret set is no more than 50, while auditing typically requires a much larger set.

That said, since the dataset used by the auditing method represents a worst-case dataset to maximize the lower bound the method can achieve, an alternative fair comparison is to use the worst-case empirical privacy measures, as suggested by the reviewer. Since we still observe a low correlation between empirical privacy and auditing scores, this suggests the choice of dataset is likely not an influencing factor.

We believe both hypotheses we made in the “Open questions” in Sec. 5.1, 1) The auditing method is not sufficiently powerful; 2) There is a fundamental gap between membership inference and memorization-based attacks (which we also bring up in the previous point), are likely; probably even the mix of both.

Q1 - one-to-many relationship. $\sigma$ is not chosen to be the minimal per-step noise but rather dependent on $b$ and $T$. It is computed using privacy accountants to satisfy a target $(\varepsilon,\delta)$-DP guarantee (Sec 2).

For Laplace or Gaussian mechanism with a fixed $\delta$, the privacy budget $\varepsilon$ uniquely determines the privacy level via the single hyperparameter that controls the noise scale. In contrast, the compositional nature of DP-SGD allows infinitely many configurations to achieve the same $\varepsilon$, each yielding its own privacy level. We intend to highlight that empirical privacy variance is a unique characteristic of DP-SGD, and more broadly, of DP algorithms that involve composition.

Q2 - certification. As illustrated in Fig 4, all models in the red region (with $\varepsilon < \varepsilon^\star$) will fail to pass the privacy tests. Please refer to our last response to Reviewer u6Yf for more details.

We thank the reviewer for recognizing our claims and methods as clear, sound and relevant, and our contributions to be of broad interest to the community. We will take the reviewer’s feedback and incorporate a more detailed related work section into the main body, incorporating discussions in our response to Reviewer E9Wi. We will also include a limitation section discussing what has not been covered in our work but deserves attention: in particular, the subtle difference between shuffling and Poissong subsampling (which is currently mentioned in a footnote), as well as the privacy cost of hyperparameter tuning.

Regarding the reviewer’s comment on the generality of heuristics, we clarify that while they are developed based on regression analysis in two specific scenarios (fine-tuning GPT-2-S on Enron at $\varepsilon=4$ with the target variable as ACR, and Llama-2-7b on TOFU at $\varepsilon=16$ with the target variable as AIR), our experiments demonstrate that they generalize across various models, datasets, privacy budgets, and empirical privacy measures (VMR). The detailed results are presented in Figs. 7 and 10 in the main paper, as well as Figs. 18, 20, 21, and 22 in Appendix G. In total, we have spent over 10,000 H100 GPU hours to ensure that our findings are general and robust. We hope this convinces the reviewer about the broad applicability of these heuristics. Please let us know if you have further questions or suggestions. Thank you!

We thank the reviewer for raising the subtlety between implementing DP-SGD with shuffled batches, but performing privacy accounting as if Poisson subsampling was used.

In the camera-ready version, we are happy to expand the discussion and move it to the main paper under a “limitation” section. Additionally, we would like to clarify a few more points.

While as pointed out the exact $\\varepsilon$ values are incorrect, we believe that the qualitative observations we make are still relevant. For example, in the plots where we have $\\varepsilon$ on the x-axis e.g. Figures 2, 3 and 10, if the correct privacy accounting of shuffle-based DP-SGD were to be applied for the specific parameter settings we have, it would only re-scale the x-axis in some (non-linear) manner, but the empirical variance of the y-axes will remain the same which is precisely the main take-away from this paper. That is, the x-axis, instead of being supported on $\\varepsilon$ in $\\{1, 2, 4, 8, 16\\}$, would be supported on some other different values, though still monotonically increasing.

Thus, while the observations made by Chua et al. (2024b;c) are valuable, we consider them relevant primarily if some model is released after training on truly sensitive data, with a claimed privacy guarantee that is incorrect. But in our case, we feel this observation is not that critical, as it just amounts to some rescaling of values which do not affect the qualitative conclusions from our study. But thank you for raising this; we will elaborate more on this in the revision.

Additionally, we note that Chua et al. (2024c) was posted on arXiv on Nov 6, 2024, which is within three months of the ICML 2025 submission deadline (Jan 30, 2025). According to the ICML 2025 reviewer instructions, this qualifies as a “concurrent work” that “Authors are not expected to discuss”. In fact, while we are aware of the earlier work Chua et al. (2024b), which highlights the gap between these two sampling approaches, we were not aware of any efficient implementations of Poisson subsampling for DP fine-tuning LLMs: 1) Chua et al. (2024c) was not available online when we initiated our work; 2) They conducted experiments on MLPs only, and extending the implementation to transformer training is non-trivial and warrants further research. As a consequence, we adhere to the conventional DP fine-tuning framework for LLMs that employs shuffle-based DP-SGD.

Since this was the main concern raised in the review, we hope the reviewer could reconsider the rating if the above response adequately addresses their concern. We are happy to answer any other questions during the discussion period if it helps to clarify this.

For other minor comments--

ACR underestimates the privacy loss. In our studies, the measured empirical privacy scores depend heavily on the chosen set of secrets. The reviewer is correct that using a shorter (sub)string as the secret or not requiring an exact match could result in a lower measured privacy loss. However, the magnitude of these scores is not our primary focus; rather, our arguments center on the variance of the empirical privacy scores.

Figures. Thanks for the suggestions, we will incorporate them in the camera-ready version.

Clarifications on “deemed to meet privacy requirements”. Here's what we intend to convey: even if a legislative body determines that a model with a privacy budget $\varepsilon^\star$ passes their privacy tests, this does not imply that models with a stricter privacy budget ($\varepsilon \le \varepsilon^\star$) will pass. For instance, as illustrated in Fig 4, all models in the red region will fail to pass the privacy tests.

Thanks for the review.

Summay of our key points

• We measure empirical privacy in controlled settings
• We conduct extensive experiments (> 10,000 H100 GPU hours) for robust findings
• Our contributions go well beyond “showing the existing variance”
• We recommend empirical privacy as an additional dimension of privacy alongside DP, rather than targeting it in isolation

Detailed responses

Sources of variance. The empirical privacy variance we observe is conditioned on fixing the dataset, model, training algorithm and empirical privacy measure (Sec 3.3 para 1). We also show variance from training randomness is small (Appendix F.3).

The reviewer raised variance in measuring empirical privacy scores. We believe this is not the case for our experiments. For ACR, we run the optimization with 3 seeds and report the highest score for more accurate estimates; VMR and AIR average 10 stochastic decodings to reduce variance (Appendix E.6).

Comparison with related works. Appendix A discusses related work. Hayes et al. (2023) and Kaissis et al. (2024) show the attack success rate of a reconstruction attack (which relies on strong assumptions) can vary for mechanisms calibrated to the same DP guarantee. We focus on realistic privacy risks that emerge in language model interactions, leading to the concept of empirical privacy variance.

Beyond this, we have more substantial contributions: 1) extensive experiments across various models, datasets, secrets, and empirical privacy measures; 2) in-depth qualitative analysis of the impact of hyperparameters and investigation of the cause; and 3) broader implications: for practitioners, we provide heuristics that enhance empirical privacy; for researchers, we expose the hidden cost of hyperparameter tuning; for policy makers, we discuss how this phenomenon could complicate standardization.

As Reviewer MX8d noted, “...this is the first time empirical privacy variance is proposed and investigated on a practical level. The experiment results provide compelling evidence that the problem is general, and likely to impact a variety of models using DP-SGD… the implications of empirical privacy variance are potentially broad and of interest to the DP community.”

Given the above, we kindly ask the reviewer to reconsider the rating based on our contributions.

More empirical validation. Figs. 7 and 10 and Appendix G (Figs. 18, 20-22) confirm our heuristics’ generality across models, datasets, $\varepsilon$’s and empirical privacy measures. Could the reviewer kindly suggest additional evidence they would like to see?

Issues with ACR/VMR. While ACR has limitations, we believe a lower ACR/VMR score indicates lower memorization and thus better privacy in general. Importantly, we don’t advocate for targeting these in isolation. Our heuristics operate on configurations calibrated to a specified DP guarantee (Sec 4.3), adding a dimension not captured by DP.

Q1 - certification. We mean: even a legislative body determines that a model with $\varepsilon^\star$ passes their privacy tests, this does not imply that models with a stricter privacy budget ($\varepsilon \le \varepsilon^\star$) will pass. As illustrated in Fig 4, all models in the red region will fail.

Q2 - theory for ACR/VMR. The statement “ACR and VMR are small for DP mechanisms” is incomplete: 1) “small” is not quantitatively defined; 2) the privacy parameters of the DP mechanism are not specified. We do not have a theory for it—as we point out in the caption of Fig. 4, bridging DP and empirical privacy measures is generally hard.

The reviewer might be suggesting that without theoretical guarantees, a model with DP can have large ACR/VMR, so our findings are not surprising. But we want to emphasize: what we believe makes our finding interesting is the simultaneous existence of both high and low ACR/VMR under the same theoretical privacy budget (see Fig. 2). This naturally motivates our study in Sec. 4 to target the low end by proper hyperparameter selection.

Q3 - hypothesis in Sec 5.1. The hypothesis here is two-fold. First, while all mechanisms are calibrated to the same $(\varepsilon, \delta)$-DP, the final LLMs (where we measure empirical privacy) might have different “real” $\varepsilon$’s, all upper bounded by $\varepsilon$. Second, these “real” $\varepsilon$’s reflect empirical privacy: a higher value corresponds to poorer empirical privacy.

Q4 - why relevant. For Laplace or Gaussian mechanism, $\varepsilon$ uniquely determines the privacy level via the single hyperparameter that controls the noise scale. In contrast, the compositional nature of DP-SGD allows infinite configurations to achieve the same $\varepsilon$, each with unique privacy levels. Fig. 4 highlights that empirical privacy variance is a unique characteristic of DP-SGD, and more broadly, of DP algorithms that involve composition.

Please let us know if you have further questions. Thank you!