A Comprehensive Study of Privacy Risks in Curriculum Learning

Motivation: First, We argue that this work is well-motivated, as evidenced by the application of CL in emerging language models [1]. Researchers have proposed a CL approach termed 'length-based CL,' which uses the length of input text as a measure of difficulty.

Secondly, the study referenced in [2] investigated data ordering for data poisoning in machine learning, confirming that this concept is more than just a hypoyhesis. While “data ordering” might seem vague, its definition in the context of CL is well-defined— it refers to the sequence in which data are presented to the model. This definition aligns with that in [2], lending further support to our motivation for investigating the potential negative impacts of such specific data ordering. Regarding your comment about 'which data points will be affected the most’, identifying these is one of the goals of our study. We began with the overarching question of how CL influences model privacy in general and then proceeded to a sample-wise investigation to determine which types of data points are most susceptible.

Structure and Technical details: See official comment on top please.

Analysis: We argue that our results regarding AIA, as shown in Figure 6 (which will be updated to Table 3) and Figure 15, demonstrate both overall and sample-wise privacy investigations on AIA. The fact that AIA does not indicate increased vulnerability in CL simply suggests that CL does not fundamentally alter how the target model captures irrelevant (or, in our specific example, sensitive) attributes.

Question 1: Both definitionsare recognized in curriculum learning literature[3]. We specifically use 2) and have previously tested out 1) too, our preliminary results showed that they lead to same conclusion.

I find the concept of sample occurrence very interesting and would love to add more discussion and experiment for it. However, we argue that occurrence is closely linked to difficulty; generally, a higher occurrence implies easier samples (based on your listed definition 2) of CL). Based on our current results, data ordering exerts a more significant influence on performance than the frequency of data occurrence. For example, in normal training (as shown in Figure 1), hard samples appear in all epochs, while in Curriculum Learning (bootstrapping and transfer learning), there hard samples appear less especially in the early epochs. Nonetheless, the MIA accuracy for hard samples in CL (e.g., samples with difficulty level of nine) still surpasses that in normal training.

To further explore the impact of occurrence, especially on easier samples which present a less intuitive case, we plan to train a CL model with disjoint batches. This approach will help us isolate and assess the effect of occurrence. It's important to note that our preliminary results indicate that using disjoint batches leads to the same key conclusions as presented in this paper. This implies that the occurrence is not the primary factor causing vulnerability in CL; rather, it is the data ordering. Samilar conclusion can be drawn from Figure 1 for the hard samples (see last paragraph).

Question2: We're interested in understanding why you find the concept of 'harder samples having a lower attack success rate in normal training' counter-intuitive. Could you please elaborate on that? To explain, let's first define what makes a sample 'difficult.' A common measure of difficulty is the loss score of a sample, where a higher loss score indicates greater difficulty. In the context of transfer learning, this score is derived from a pre-trained model. A high loss score could mean that the pre-trained model struggles to 'remember' this sample, possibly because the sample is rare, its features are too distinctive, or for other reasons. If a target model has difficulty remembering a sample, it's logical to assume that such a sample would be less vulnerable to attacks, as the model reveals less information about it.

We also want to clarify that when we say 'harder samples are more vulnerable,' we specifically refer to their vulnerability in the context of CL compared to a non-CL setting. This might be more an issue of how we present our ideas. Phrasing it as 'CL introduces stronger effects on harder samples compared to normal training' seems to convey the same idea in my opinion. However, we're open to rephrasing this for greater clarity for the audience.

[1] Length-Based Curriculum Learning for Efficient Pre-training of Language Models. New Gen. Comput. 41, 1 (Mar 2023)

[2] Manipulating sgd with data ordering attacks. Advances in Neural Information Processing Systems.

[3] Petru Soviany, Radu Tudor Ionescu, Paolo Rota, Nicu Sebe: Curriculum Learning: A Survey.

Weakness 1: We can possibly add text classification task upon request with data set from torchtext.datasets to this study to make it more comprehensive. More specifically, we plan to use the example shown in [1] for this study.

Weakness 2: In general, many of these defenses operate under different assumptions and vary in effectiveness depending on the setting.

DP-SGD trains models with noise, which fundamentally reduces the model’s ability to memorize individual samples, thus leading to superior defensive performance.

MemGuard, on the other hand, alters a model’s output to defend against MIA. This is the reason that it doesn’t affect the target model yet can still be effective. However, it also does not offer protection against label-only attacks due to its lack of changes to the target model.

MixupMMD and AdvReg both modify the posterior to enhance privacy. MixupMMD is more effective because it brings the member and non-member posterior distributions even closer than AdvReg by introducing the new penalty termed Maximum Mean Discrepancy.

From all the defenses mentioned above, we've learned that a key factor in making a defense effective is to minimize the difference between member and non-member posterior distributions. Therefore, a potential direction for future research in improving defenses could involve exploring other methods to further align these posterior distributions. We plan do add this discussion to the paper.

Question 1: Could you please further elaborate on this question? If we understand your question correctly, are you asking whether modifying the difficulty score can make Curriculum Learning (CL) less vulnerable to attacks, and whether the difficulty score can be utilized as part of a defense strategy? Based on our study across various CL settings (e.g., baseline), we found that even with randomized difficulty scores, CL remains more vulnerable compared to normal training. Therefore, the key to a more robust defense lies in minimizing the gap between member and non-member posterior distributions.

Question 2: We consider federated learning and decentralized training a parallel line of study to ours. However, when combined with our study, the key conclusions remain the same for all decentralized models, and we believe the central model will be less intrusive compared to our findings. This is because, fundamentally, federated learning and decentralized training are designed to offer enhanced privacy.

[1] https://github.com/pytorch/text/tree/main/examples/text_classification

Weakness and Question: We argue that even though our choice of tools and Curriculum Learning (CL) approaches are based on existing works, we are the first to study the privacy risks introduced by CL. Furthermore, we believe the prominence of privacy concerns is more convincingly demonstrated if existing MIA can already cause privacy issues in CL, not to mention the potential impact of newer attack methods. We found that Diff-Cali is extremely effective in terms of True Positive Rate (TPR) at low False Positive Rate (FPR), achieving over 50% for all samples compared to the NN-based attack. This effectiveness is significant, considering the importance of TPR at low FPR as a metric. As [1] states, “This focus (on the low-false positive rate regime) is the setting with the most practical consequences. For example, to extract training data, it is far more important for attacks to have a low false positive rate than a high average success rate, as false positives are far more costly than false negatives. Similarly, de-identifying even a few users contained in a sensitive dataset is far more significant than making an average-case statement like ‘most people are probably not contained in the sensitive dataset’”.

Minor: The entry in Table 2 contains a typographical error and should read 0.8577; additionally, Figure 6 has been changed to Table 3.

[1] “Membership Inference Attacks From First Principles” in IEEE S&P 2022

W1-LiRA: In general, not all attack methods (e.g., adaptive attack and LiRA) and defense techniques have been examined in our paper. Although LiRA is considered state-of-the-art, it requires lots of shadow models (for example, 256 shadow models for CIFAR100, as demonstrated in [1]), whereas all other attacks in our paper require only one. To compare fairly with LiRA, we would need to divide the current datasets into much smaller subsets, which would likely degrade the performance of all target and shadow models. Moreover, training 256 shadow models for both CL and no-CL settings would be very costly. Due to these reasons, LiRA was not investigated in this study. However, we believe our key findings (e.g., the increased vulnerability of difficult samples when trained with CL) are broadly applicable, given the fundamental principles of curriculum design.

Furthermore, we can possibly run a downscaled version of LiRA upon request (i.e., with fewer shadow models) in a case study to demonstrate that our key conclusions remain valid. More specifically, we intend to re-divide the dataset into six subsets, $D_i$ where $i = 0,1,2,3,4,5$. We will use $D_0$ to train the target model, $D_1$, $D_2$, $D_3$, and $D_4$ to train shadow models for LiRA and NN-based methods, and then use $D_5$ as the target and shadow test dataset.

W2: From the technical perspective, Diff-Cali only made a small change. However, its performance proves to be extremely effective in bringing the True Positive Rate (TPR) at a low False Positive Rate (FPR) above 50%. Thus, we argue that it is effective, even though we didn’t make any major changes. Furthermore, we argue that TPR at low FPR is an extremely important metric. Quoting from [1], “This focus (on the low-false positive rate regime) is the setting with the most practical consequences. For example, to extract training data, it is far more important for attacks to have a low false positive rate than a high average success rate, as false positives are far more costly than false negatives. Similarly, de-identifying even a few users contained in a sensitive dataset is far more significant than making an average-case statement like ‘most people are probably not contained in the sensitive dataset’”.

W3: We tested Diff-Cali in both non-CL settings and various CL settings (refer to Figures 3 and 5). In these figures, we used bootstrapping as the default curriculum referenced in Diff-Cali and then applied this attack to models trained with all non-CL and CL methods (the "normal" lines in Figures 3 and 5 represent the non-CL setting). As observed, difficult samples become more vulnerable in all cases, regardless of whether CL is used in the target model. However, when the matching CL is applied, the improvement in accuracy on difficult samples is more pronounced. For instance, comparing Figure 1a with Figure 5a, the improvement in accuracy at difficulty level 9 is approximately 2.5 for anti-CL, while for bootstrapping, it is nearly 7.5.

Question-Defense: We have not yet explored this, but we believe the fundamental conclusions would remain unchanged, given DP-SGD's proven success in defending against MIA. Allocating the privacy budget based on the difficulty level could potentially yield a better balance between privacy and utility. However, this approach would impact the moments accountant technique used by DP-SGD for total budget accounting and require an examination of privacy composition. Additionally, incorporating the difficulty level might alter the privacy guarantee, possibly requiring the addition of noise to this difficulty level as well.

[1] “Membership Inference Attacks From First Principles” in IEEE S&P 2022