Curvature Clues: Decoding Deep Learning Privacy with Input Loss Curvature

We thank the reviewer for their feedback, we address the weaknesses and questions below:

1. Missing some references, e.g. Li et al. 2023 or the original DetectGPT paper (Mitchell et al. 2023).

A: We thank the reviewer for the feedback, we will include these references in the revised version of the paper.

2. I believe the paper would benefit from including some sort of proof sketch of the theorems in the main body.

A: We thank the reviewer for the feedback, we will include a brief sketch of proof in the main body of the paper in the revised version of the paper.

3. Other authors have used the Hutchinson trace estimator to estimate information about input loss curvature. How does the performance of this method compare to the zero-order method presented in the paper?

A: We provide additional results on the effect of zero-order estimation. We run the proposed MIA attack on CIFAR100 and CIFAR10 datasets with zero order (ZO) estimation (black-box) and compare it against white-box Hutchinson curvature calculation [Garg et al.]. The results are provided in Table 1 and Figure 3 in the rebuttal pdf (please see the global response). Please note this comparison is a black-box (ours) vs white-box (Hutchinson) comparison hence is provided for a complete analysis.

We thank the reviewer for their feedback, we address the weaknesses and questions below:

1. What is $s_f$, $L_f$ and $c_f$ in Line 300? The notation seems to be missing from the context. Line 117 missing a .

A: $s_f$, $L_f$ and $c_f$ are the fitting coefficients for equation 8, we will clarify this in the revised version of the paper and include missing punctuation in line 117.

2. How accurate is the zeroth-order estimation of input loss curvature? Can you provide a comparison of this estimation to the real values (assuming we have white-box access to target model)

A: We provide additional results on the effect of zero-order estimation. We run the proposed MIA attack on CIFAR100 and CIFAR10 datasets with zero order (ZO) estimation (black-box) and compare it against white-box Hutchinson curvature calculation [Garg et al.]. The results are provided in Table 1 and Figure 3 in the rebuttal pdf (see global response). Please note this comparison is a black-box (ours) vs white-box (Garg et al) comparison hence is provided for a complete analysis. Additionally, please see our global response.

We thank the reviewer for their feedback, we address the weaknesses and questions below:

1. In the theoretical analysis, the authors propose the train-test distinguishability distribution upper bound for both data and curvature. However, they compare the upper bounds directly and suggest that curvature based MIA is a better method when the size of dataset is large enough. The comparison between upper bounds may not reflect the real relationship. To validate this result, one may compare between an upper bound and a lower bound.

A: We agree with the reviewer that comparing the upper bound with a lower bound would be a more elegant approach. However, to account for the fact the we compare upper bounds, we provide empirical evidence to validate this result in Figure 6 (please see the updated plot in the rebuttal pdf, specifically Figures 1 and 2). Additionally, please see the global response bullet 2 and the pdf with revised dataset size vs performance plots validating the theoretical result. In summary, we use empirical results to bolster and check the validity of the theoretical result. The results show we perform better than other methods and as predicted by theory.

2. How the estimated curvature deviates Gaussian and can we still model the distribution with it?

A: The validity of the Gaussian assumption is difficult to assess using a single plot, such as Figure 3 in the main paper, which depicts one image from ImageNet. It was provided as a visual aid for the reader. Parametric MIA methods demonstrate the 'goodness' of parametric estimation in the low FPR region, as shown by Carlini et al. Specifically, accurate parametric estimation results in good low FPR performance [Carlini et al]. Our results in Figure 4 and Table 1 empirically validate that the Gaussian assumption is a good parametric estimate. That is we see the Curv LR method which is the parametric model outperforms the Curv NLL method and others in low FPR performance.

3. What is the effect of the error in the zeroth-order curvature estimation?

A: We provide additional results on the effect of zero-order estimation. We run the proposed MIA attack on CIFAR100 and CIFAR10 datasets with zero order (ZO) estimation (black-box) and compare it against white-box Hutchinson curvature calculation [Garg et al.]. The results are provided in Table 1 and Figure 3 in the rebuttal pdf. Please note this comparison is a black-box and white-box comparison hence is provided to illustrate the effect of the error in the zeroth-order curvature estimation.

We thank the reviewer for their feedback, we address the weaknesses and questions below:

1. While the loss value based MIA evaluation generalizes to pretty much all class of models, it’s unclear if loss curvature based MIA attack would enable a similar generalization, i.e., going beyond the small-scale classification setup to self-supervised models, contrastive models, and other foundation models [1].

A: The proposed curvature score is generic and only requires a valid loss function to measure model performance. Therefore, it can theoretically be applied to any machine learning model. The only limitation is with auto-regressive models, where defining input loss curvature is non-trivial. To test the generalizability of the method, we evaluated it across additional architectures. The results are presented in Table 3 in the rebuttal PDF. We appreciate the reviewer's input and will revise the paper to include these additional results.

2. As fig. 6 itself highlights, the robustness of attack against a simple defense that subsamples the training data for shadow models is very low. Just using ½ of the training data renders the attack ineffective (albeit Carlini et al. suffers from the same limitation). Since shadow models are only a proxy of the true black-box model with training data information not publicly available, in practice they will likely only be trained on a subset of original training data of the block-box model, thus significantly reducing the attack success in black-box threat model.

A: Further investigation and code audit revealed that the performance degradation at 50% dataset size in Figure 6 was caused by a bug in the experiment. This one-line bug (see the diff in the rebuttal pdf) resulted in under-representing the membership inference attack (MIA) performance of our method and that of Carlini et al. After fixing this bug, the updated graph for Figure 6 in the rebuttal PDF (Figures 1 and 2 in the global response pdf) shows that the curvature-based method performs better than before. Specifically, as predicted by Theorem 4.3 beyond 20 − 30% (Figure 2 and 1 in the rebuttal pdf) subset Curv ZO LR matches or outperforms prior works in AUROC (better in TPR at low FPR) and Curv ZO NLL (Figure 1 in the rebuttal pdf) outperforms prior works beyond 10% subset size (compared to 40% previously). Both our method and Carlini et al.'s do not show a performance drop at 50% dataset size. We thank the reviewer for their input and will revise the paper to include these additional results.

3. Why not use zero-order loss estimation in black-box method as it would be significantly more sample efficient than block-box curvature estimation.

A: This is indeed possible; however, our empirical results suggest curvature outperforms loss-based attack. We use the true loss (not estimated) for LOSS attack [as described Carlini et al.], which performs poorer than LIRA [Carilli et al] and our method (supported by Table 1 and Figure 4 in the main paper, and Table 2 in the rebuttal pdf). The main reason being loss-based MIA is non-parametric. Non-parametric MIA methods struggle at low FPR performance (also shown by Carlini et al.). Since the loss distribution is non-gaussian Carlini et al. suggest logit scaling to build parametric MIA i.e. LiRA. Our experimental results show that we outperform both of these methods (Carlini et al. and LOSS).

4. Would using the first-order statistics, gradients of the loss w.r.t input, be as helpful as curvature in MIA attacks?

A: Indeed, first and zero-order statistics can be useful. To evaluate this, we compare these metrics to the memorization score from Feldman and Zhang, which is an independent metric of memorization in deep models. We find that curvature aligns best with memorization at the end of training and thus outperforms zero and first-order (i.e., loss and loss gradient) metrics. See the Table 4 in the rebuttal pdf. Table 4 shows the cosine similarity of Feldman Zhang (FZ) memorization scores (independent metric) with the different order statistics, for all the samples in CIFAR100 and top 5000 most memorized samples in CIFAR100 (according to Feldman and Zhang). Further, the loss-based attack [LOSS from Carlini et al.] is provided as reference in Table 2 (rebuttal pdf) which fails to outperform curvature. We thank the reviewer for their input and will revise the paper to include these additional results.

5. However .... Thus it’s critical to ablate the choice of network architecture and demonstrate that despite the diversity in architectural designs, the approach succeeds in MIA attacks. A few simple variations would be changing the normalization layers, activation functions, and broadly the class of models (CNNs vs transformers).

A: We consider the second derivative of the loss with respect to the input. Curvature estimation involves sampling the loss landscape at specific points to determine curvature. This method assumes that the function is smooth. If the function is piecewise linear, the zero-order (zo) method estimates the curvature of the smoothest function that intersects all sampled points, thereby smoothing the piecewise linear function. This approach is validated by our membership inference attack (MIA) results across various architectures, including ResNet18 with batch normalization and ReLU (as detailed in the main paper), and additional results for VGG11, VGG11_BN, Inception, and Vision Transformers (ViT-B16) as presented in the rebuttal document. The results demonstrate that zo curvature outperforms prior methods (see Table 3 in rebuttal PDF).

Feldman, V. and Zhang, C. What neural networks memorize and why: Discovering the long tail via influence estimation. Advances in Neural Information Processing Systems, 33:2881–2891, 2020.

N. Carlini, S. Chien, M. Nasr, S. Song, A. Terzis, and F. Tramer. Membership inference attacks from first principles. In 2022 IEEE Symposium on Security and Privacy (SP), pages 1897–1914. IEEE, 2022.