Defending Membership Inference Attacks via Privacy-aware Sparsity Tuning

1.Lack of Comparison with Recent Baselines: The current version does not compare its approach with recent baselines from the past three years, which is crucial for an ICLR 2025 submission. The paper only references MIA attacks from six years ago and defenses from three years ago, missing key recent works that are important for establishing state-of-the-art (SOTA) context. Key references to consider include:

MIA Attacks:

(a)"Low-Cost High-Power Membership Inference Attacks" (ICML 2024)

(b)"Scalable Membership Inference Attacks via Quantile Regression" (NeurIPS 2023)

(c)"Membership Inference Attacks from First Principles" (S&P 2022)

(d)"Enhanced Membership Inference Attacks against Machine Learning Models" (CCS 2022)

MIA Defenses:

(a) "MIST: Defending Against Membership Inference Attacks Through Membership-Invariant Subspace Training" (USENIX 2024)

(b) "SELENA: Mitigating Membership Inference Attacks by Self-Distillation Through a Novel Ensemble Architecture" (USENIX 2022)

These references represent a minimum set of recent works that should be discussed and included as baselines for a credible comparison.

2.Method Simplicity and Lack of Theoretical Rigor: The proposed method relies on a heuristic reweighting approach that lacks theoretical guarantees, which is a notable limitation for top conference standards. The reweighting method in Privacy-Aware Sparsity Tuning (PAST) is too simplistic and does not reach the research depth, completeness, and robustness generally expected of accepted MIA papers in top conferences. The authors are encouraged to enhance the theoretical foundation and complexity of their approach for future submissions.

3.Weak Motivation and Contribution: The paper's motivation, as shown in Figure 1(b) ("Only a small fraction of parameters substantially impacts privacy risk"), is already a well-known concept in machine learning. Given that parameters closer to the output layer naturally have more influence on gradients and results, this finding is not novel. The use of regularization as an MIA defense is also limited, as regularization is not a commonly adopted SOTA defense method in MIA. The paper does not address whether the proposed reweighting improvement would benefit more commonly used defenses, making its contribution less impactful given that it builds on a sub-optimal defense method.

There are flaws and issues in the motivation of considering the loss gap, the consistency between the attacker advantage and the loss gap as the functional proxy of privacy risk, the experimental setup, and the characterizations related to the privacy-utility tradeoff.

I would be open to increasing the score if the authors can adequately address my concerns during the rebuttal.

Comment 1:

The rationale for using the loss gap as a proxy risk for privacy is unclear. Specifically, the referenced paper (Sablayrolles et al., 2019) considers a specific case in which the posterior in Eq. (1) is modeled as a Boltzmann distribution, creating a direct link between the loss function and the posterior distribution. Generally, the posterior distribution is crucial in characterizing privacy risk. The work by Sablayrolles et al. (2019) assumes a direct connection between the loss function and privacy risk, with all results depending on this assumption, which does not extend to general loss functions. Therefore, using the statement, “It has been shown that the loss function can determine the optimal attacks in membership inference (Sablayrolles et al., 2019),” as justification for adopting loss gaps as a proxy for privacy loss is questionable and insufficient. In addition, the paper does not theoretically or analytically characterize the rationale for using the loss gaps as a proxy of the privacy risk in terms of attacker advantage.

In experiments, three classes of MIA models are considered. However, the use of the loss gap as a proxy for privacy risk does not fully coincide with the privacy risk induced by these MIA models. In particular, the loss gap captures the average loss distinguishability between the member and the non-member. However, different MIA models including the three considered use a range of observations, including logits, confidence, entropy, correctness, and augmented predictions. These observations (of the attacker) capture different aspects of the mechanism's/model's output and behavior beyond just the loss values. As a result, the privacy risk represented by these attacks may involve patterns in these observations that a simple loss gap does not fully capture. For example, the augmentation-based attack depends on how the model responds to augmented inputs, which can reveal membership through consistency in the model's behavior across transformations. Loss gap does not capture this aspect of the model's sensitivity to augmented variations, so it might underestimate privacy risk in such scenarios.

Thus, it is in general possible for two different weight schemes with similar loss gaps to exhibit notably different attacker advantages.

Comment 2:

My concern regarding the feasibility of using loss gap as a proxy of privacy risk leads to the concern of adaptively setting the parameter weights based on the privacy sensitivity (i.e., the gradient of the loss gap w.r.t. the parameter). That is, the gradient might not precisely reflect the privacy sensitivity. It is suggested to provide analytical and theoretical characterizations for the feasibility of using loss gap as a proxy of privacy risk, including the possible assumptions.

Comment 3:

The comparison between the PAST and the uniform-penality-based $\ell_1$ and $\ell_2$ baselines (or uniform baselines) needs more details. The paper clearly describes the difference between PAST and uniform baselines in terms of whether the weights of the parameters are uniform or privacy-aware. However, the impact of the magnitude of the weights on privacy risk is ignored (correct me if I am missing anything).

Q1: In Figure 3 and Figure 4, what are the values of lambdas chosen for $\ell_1$ and $\ell_2$ baselines?

Q2: Is it possible that the performance differences between PAST and the $\ell_1$ and $\ell_2$ baselines shown in Figure 3 and Figure 4 are due to the combination of the magnitude of the weights and setting of uniform or privacy-aware penalties, rather than due to the setting of uniform penalties or privacy-aware alone?

It is intuitive that properly weighting the more-privacy-sensitive parameters more than the less-privacy-sensitive parameters might be enough to guarantee a certain degree of privacy. However, the comparison of PAST and the $\ell_1$ and $\ell_2$ baselines in terms of the privacy-utility/accuracy tradeoff without considering the magnitude of the weights could lead to unfair comparisons.

Comment 4:

Overparameterization is not the same as non-properly weighting a fixed number of parameters. The proposed PAST applies adaptive regularization by adjusting the penalty weights of parameters based on their privacy sensitivity, which does not necessarily address the issue of overparameterization.

Section 5 RELATED WORK discusses the relationship between the number of parameters (or model complexity) and privacy risk, particularly in the context of overparameterization. There are works that confirm the tradeoff between the number of parameters and privacy risk. However, there is limited discussion of how the specific values or magnitudes of those parameter weights might impact privacy. Adjusting the weights of parameters for fixed parameters could influence the model's tendency to memorize or generalize, which potentially affects privacy risk. However, the related work section does not discuss studies that study this trade-off between parameter weight and privacy directly. In addition, the paper is suggested to characterize this weight-privacy trade-off.

While I agree that in general the average loss of members and non-members are different, shrinking this gap with the proposed sparsity technique can be effective to those loss-based (or related) MIAs. But it may not be able to defense against other more advanced state of the art MIAs. Here are my two main concerns:

1. The evaluation metric for privacy is out of dated, this metric has been criticized in many recent MIAs [R1] D. Hintersdorf, L. Struppek, and K. Kersting, “To trust or not to trust prediction scores for membership inference attacks,” in IJCAI, 2022 [R2] S. Rezaei and X. Liu, “On the difficulty of membership inference attacks,” in IEEE Conf. Comput. Vis. Pattern Recog. (CVPR), 2021 [R3] N. Carlini, S. Chien, M. Nasr, S. Song, A. Terzis, and F. Tram `er, “Membership inference attacks from first principles,” in IEEE Symposium on Security and Privacy (S&P). IEEE, 2022

I suggest the authors to test the performance using TPR at low FPR rates for assessing the privacy risks.

2. The evaluated MIAs are also out of dated, most of them are metric based MIAs. Their performances are very close to each other and more importantly, these attacks often produce misleading results, which have been extensively explored in [R4] Li, Xiao, et al. "On the privacy effect of data enhancement via the lens of memorization." IEEE Transactions on Information Forensics and Security (2024). [R5] Aerni, Michael, Jie Zhang, and Florian Tramèr. "Evaluations of Machine Learning Privacy Defenses are Misleading." arXiv preprint arXiv:2404.17399 (2024).

I suggest the authors to adopt more state of the art MIAs for example LiRA [R6] and RMIA [R7] because there are shown to be effective in the TPR at low FPR regions. More importantly, the results of these attacks, particularly LiRA, align consistently with the definition of privacy risk, as well as with the concepts of differential privacy and memorization degree [R4, R5]. This contrasts with the deployed metric-based MIAs, which often yield misleading results by acting as strong non-member detectors. The reason for this discrepancy is that metric-based MIAs typically exploit the fact that models are more confident in members than non-members. However, this confidence does not hold at a more nuanced level; members with high privacy risk are often difficult and atypical samples, which tend to have low confidence and are therefore frequently misclassified as non-members.
[R6] Carlini, Nicholas, et al. "Membership inference attacks from first principles." 2022 IEEE Symposium on Security and Privacy (SP). IEEE, 2022. [R7] Sajjad Zarifzadeh, Philippe Liu, and Reza Shokri. Low-cost high-power membership inference attacks, 2024. URL https://arxiv.org/abs/2312.03262

As far as I can see, authors do not consider SOTA shadow-model based attacks like LiRA (Carlini et al., 2022) and Attack R (Ye et al., 2022). It would be important to understand whether their findings hold for a strong MIA adversary capable of trainings shadow models.