Impact of Dataset Properties on Membership Inference Vulnerability of Deep Transfer Learning

We sincerely appreciate your review and useful comments. We address each comment below.

The paper needs to clarify its motivation. While it discusses transfer learning and few-shot learning, it's unclear whether these techniques are studied as potential privacy protection methods or whether the authors simply examine their privacy risks. The introduction introduces these learning approaches but doesn't explicitly position them against existing differential privacy (DP) defenses. This makes it difficult to understand whether the main goal is to propose new privacy protection strategies or to analyze privacy vulnerabilities in these methods.

Our goal is to analyze MIA vulnerability in the fine-tuning of deep neural networks. Using few-show learning itself is not for privacy protection. Rather, we focus on the fine-tuning setting because it is widely used in practical applications. In addition, fine-tuning is also useful when a large dataset of labeled examples is not available, which would be the case for privacy-sensitive applications. We clarified the introduction in the revised version.

The paper's organization should be reversed. It would be clearer to show the experimental results first and then explain the theory behind them rather than the other way around.

We organized the paper in this way because theoretical analysis presented in Sec. 3 entails more general results than the empirical results in Sec. 4.

The authors need to revise their claimed contributions. For example, their first contribution about analyzing privacy attacks at the individual sample level isn't new - other researchers have already done similar work (such as [1], [2]).

Thank you for pointing out this. The first contribution did not mean that we focused on sample-level privacy risk, but rather we derived the closed-form LiRA/RMIA vulnerability at individual sample level. We clarified the sentence in the revised version.

It is unsurprising that larger datasets exhibit lower overall vulnerability to MIA, as models trained on larger datasets tend to rely less on individual samples, reducing the risk of memorization.

As the reviewer suggests, it is indeed unsurprising that the MIA vulnerability decreases as the number of examples per class increases. However, the rate of change in this trend is not trivial. To the best of our knowledge, there is no prior work that identified the explicit rate (i.e. power-law relationship in our work). The power-law has a pragmatically useful implication that a practitioner could reduce and estimate the MIA vulnerability of a model by adding more examples in the training set.

The authors predicted the overall dataset vulnerability using a regression model with properties such as the number of classes and examples per class. While it offers useful insights, individual-level vulnerabilities are often more actionable for user privacy protections. It's much more meaningful to investigate the per-sample vulnerabilities. However, considering varying dataset sizes, it may be challenging to derive consistent per-sample results, as the same sample could behave as an outlier in some datasets but not in others.

We agree that per-sample privacy risk should be a focused concern for practical applicability. In the simplified model (Sec. 3.4), the per-example MIA vulnerability also follows the power-law relationship with respect to the number of examples per class. That is, the rate of decrease in the vulnerabilities of individual examples, including outliers, would also follow the power-law.

Thank you for your time and effort to review our paper and provide insightful feedback. We address each comment below.

The authors indicate that the paper considers two types of MIAs. There have been a wide variety of MIAs examined in the literature, including some which may not rely on confidence scores (e.g., label-based MIAs). Is there a claim (perhaps speculative) that the authors can make about similar relationships for other classes of MIAs?

As the reviewer suggests, we only focus on score-based MIAs, while there are other threat models. We are not claiming that the observed power-law relationship would hold for white-box attacks. However, we emphasize that score-based MIAs are a realistic strong threat model. In addition, for other weaker attacks, including label-only MIAs, it is sufficient to have the power-law for the currently known strongest MIAs (LiRA and RMIA) which provides the upper bound for the vulnerability to other weaker MIAs.

The experiments focus on datasets and models related to image classification. Can the results be extended to other domains, such as audio or video? Multi-modality can strengthen the scope of the proposed contribution.

Image classification has been widely studied in privacy research. We believe that limiting the experiments to this simple setting would sufficiently support our theoretical findings. However, as the reviewer suggests, it would be important to expand the scope of application domains in future research.

The authors point out that analyses and experimental results rely on an assumption that the datasets are balanced. Some perspective on what might happen if this is not the case can shed more light on real-world applicability of the work.

If the power-law relationship we identified in this work holds in a real-world application, for unbalanced datasets, a practitioner could reduce and estimate an upper-bound for the MIA vulnerability of the entire dataset by increasing the number of examples in the smallest class. Although theoretically the MIA vulnerability should not depend on the number of classes, our empirical results suggest that there could be some dependence. Therefore, when the number of classes is large, a practitioner would need to add more examples than in the case of a small number of classes to achieve the same privacy protection. We added this discussion in Sec. 5 of the revised version.

A more detailed discussion or comparison of the effect of dataset characteristics on other aspects of privacy- e.g., differential privacy, which has an extensive literature and supported by theoretical guarantees- will strengthen the submission. Could the authors comment on this?

Prior works studied how datasets with different properties (e.g. the number of examples per class) are affected by applying DP, for example. We have added references to the related work paragraph in Sec. 1. However, these prior works on DP susceptibility do not consider the rate of change in the susceptibility as the dataset properties change. On the other hand, our focus is on the explicit relationship between the privacy risk and dataset properties (i.e. power-law) in the non-DP setting. To the best of our knowledge, there is no prior work that explicitly established the effect of dataset properties such as the number of examples per class on MIA vulnerability or DP susceptibility.

We sincerely appreciate your review and very constructive feedback. We address each point below.

Studying a well established problem, while not taking prior work into account

Thank you for pointing out relevant papers. Song & Mittal (2021) empirically demonstrate the positive correlation between the privacy risk and the generalization error of each class. However, Song & Mittal (2021) define the privacy risk of a target example as the posterior probability of the target being in the training set given the attacker's observation.Thus, their findings are not directly comparable to our MIA vulnerability metric TPR at a low FPR. Yeom et al. (2018) theoretically and empirically show similar results. Nonetheless, Yeom et al. (2018) evaluates MIA performance by maximizing TPR - FPR, not for a low FPR. Therefore, this is also not comparable to our findings. In addition, both of these two papers are not concerned with dataset properties that our work studies, fixing the dataset size and the number of classes. Although the number of examples per class explored in our work would indeed be related to the model's generalization, Yeom et al. (2018) in fact demonstrate that generalization error is not necessary for MIA success. We have added these references to the related work paragraph in Sec. 1.

Suriyakumar et al. (2021) empirically show that classes with less examples have less influence on prediction performance when DP is applied. Bagdasaryan et al. (2019) demonstrate similar results that smaller subgroups are more strongly affected by DP in terms of prediction accuracy. Kulynych et al. (2022) show that there are greater disparities in MIA vulnerability among subgroups of training data when the subgroup size is smaller. Although these works suggest that the number of data points in a class is indeed related to the privacy susceptibility of that class, they do not mention the rate of changes as the number of examples in a class changes. Moreover, since they do not consider the privacy vulnerability at a low FPR, the findings are not directly comparable to our work. We have added these references to the related work paragraph in the revised version.

the results are presented "as is", i.e., without embedding them into any broader context and without generalization. This makes it questionable, how well the results will age, i.e., how much can we benefit from them in 2-3 years, when LiRA and RMIA are obsolete and have been replaced by new and more powerful attacks.

We emphasize that score-based attacks are a strong realistic threat model, and LiRA with infinitely many shadow models is the optimal attack against the simplified model (Sec 3.4) by the Neyman-Pearson lemma. We clarified this in the revised version. Nevertheless, we acknowledge the limitation that there could be stronger attacks in the future for which the vulnerability behaves differently. We revised the limitation paragraph in Sec. 5.

“Hence we can have the LiRA vulnerability arbitrarily close to zero in this non-DP setting by simply increasing S” -> this seems to mean that if every class has more and more train data, the membership risk decreases. Formulating such intermediate “checkpoints” in plain English would be genuinely helpful to position the results.

The expression is revised. Thank you for your suggestion.

Evaluation done for fine-tuning might skew the results

We focused on fine-tuning of neural networks because it is widely used in practical applications. Fine-tuning is also important when a large number of labeled examples are not available in the application domain, which would often be the case for privacy-sensitive applications. Moreover, from-scratch training has been extensively studied, while research on the privacy risk of fine-tuned models remains relatively scarce. That said, we acknowledge that models trained from scratch are likely to be more vulnerable, but this is out of scope in our work. Additionally, our simplified model (Sec. 3.4.) is designed to resemble the linear (HEAD) classifier often used for fine-tuning.

Thank you for your time and effort to review our manuscript and provide insightful feedback. We address the comments below.

The authors need to clarify the differences in membership inference vulnerability between standard image classification models and fine-tuned models

Fine-tuning is increasingly used in practical applications. It is also important when a large dataset of labeled examples is not available in the application domain, which would often be the case for privacy-sensitive applications. Moreover, while from-scratch training has been extensively studied, research on the privacy risk of fine-tuned models remains relatively scarce. Therefore, we focus on the fine-tuned models, leaving from-scratch training out of scope. We now clarify this motivation in the revised version.

As the authors mention in the introduction, prior work has already explored the impact of the number of examples per class (shots) on MIA vulnerability. How does this study differ from previous work, and what new insights does it provide beyond existing findings?

This work differs from prior work in that we identify the rate of the impact, namely, the power-law relationship. The literature suggests that, if we have more data in the training set, then the model is less vulnerable to MIA. However, to the best of our knowledge prior works did not explicitly identify how much the MIA vulnerability decreases, as the size of the training dataset grows. With the evidence of the power-law relationship between MIA vulnerability and the number of examples per class, machine learning practitioners or analysts would be able to estimate the membership privacy risk and how large a training set would be needed to mitigate the risk. We clarified this in the revised introduction.

While LiRA and RMIA represent state-of-the-art approaches in MIA, it is also essential to assess the generalizability of the presented results to other MIA techniques. How might other methods differ in vulnerability patterns under the same dataset conditions?

In this work we focused on LiRA and RMIA that are state-of-the-art score-based MIAs. For white-box attacks, the vulnerability might behave differently. However, score-based attacks are a more realistic threat model, thus being our focus in this work. For other existing weaker attacks, including label-only attacks, it is sufficient to have the power-law relationship for the currently known strongest attacks as an upper bound. For potential future MIA, we acknowledge the limitation that there might be stronger attacks in the future such that the vulnerability behaves differently. That said, it should be noted that LiRA with infinitely many shadow models is the optimal attack against the simplified model (Sec. 3.4) by the Neyman-Pearson lemma. Thus, as long as the simplified model reflects real-world models, the power-law is expected to be observed. We now clearly state the optimality of the simplified model in the introduction as well as in Sec. 3.4.

As a widely used method for privacy protection, it is necessary to discuss the impact of differential privacy on the arguments presented in this paper.

Our work investigates the MIA vulnerability (TPR at a law FPR) in non-DP settings. Applying DP gives an upper bound on MIA vulnerability, and thus we would not need to worry about dataset properties. However, this comes at a cost. Therefore, it is important to understand when DP is necessary. We stress this point in the introduction.