Posterior Probability-Based Label Recovery Attack in Federated Learning

Dear authors,

I realize my review is voluminous and, therefore, takes a while to answer. However, as the end of the review period is coming really soon, I would encourage you to post a partial answer to my review so that we can engage in discussion while finishing the rest of your response.

Thank you, Reviewer gCmo

In the interest of providing the authors with quick feedback before the end of the discussion period. I have skimmed their responses and the new pdf without going through it in detail yet. My feedback follows:

Response to Q1: I am thankful to the authors. This indeed drastically helps the readability of the paper. While I haven't yet looked through all derivations from scratch, one thing I want to point out/ask about is interpreting $p_t$ as the softmax at the start of the proof on page 13. I thought, it could also be one minus the softmax instead, depending on the ground truth label. Can the authors elaborate on this/fix the derivations?

Response to Q2: I am sure the authors should be able to find an intuitive interpretation within the exponential family for the focal loss ( given its closeness to CE ) and equal vulnerability. A quick google search suggested the following article [1]. Again, due to the interest of time, I haven't gone through it in detail to see if it exactly relevant to the author's use case.

Response to Q3: In Appendix A.4. the authors discuss the computational savings of the gradient function of the softmax+CE combo. Are those savings realized by the backward pass of, say, pytorch? Are the savings significant for the training time? Can the authors give an example of another exponential family member's final gradient and how it does not have the same computation savings (both in terms of the formula and computation complexity)?

[1] https://towardsdatascience.com/cross-entropy-classification-losses-no-math-few-stories-lots-of-intuition-d56f8c7f06b0

Q8: Can the authors provide a comparison with [3] or [5] when auxiliary data is used for estimating their parameters?

We name ZLG for [3] and LLG for [5]. The following experiments compare the label recovery accuracies with different model architectures, batch sizes and activation functions.

• Batch size=64, activation=ReLU:

  Dataset	Model	ZLG ClsAcc	ZLG InsAcc	LLG ClsAcc	LLG InsAcc	Our ClsAcc	Our InsAcc
  MNIST	LeNet	1.000	0.996	1.000	1.000	0.995	0.955
  CIFAR-10	ResNet-18	1.000	0.916	1.000	0.914	1.000	1.000
  CIFAR-100	ResNet-50	0.985	0.893	0.816	0.633	1.000	1.000

• Batch size=256, activation=ReLU:

  Dataset	Model	ZLG ClsAcc	ZLG InsAcc	LLG ClsAcc	LLG InsAcc	Our ClsAcc	Our InsAcc
  MNIST	LeNet	1.000	0.977	1.000	0.982	1.000	0.951
  CIFAR-10	ResNet-18	1.000	0.905	1.000	0.919	1.000	0.977
  CIFAR-100	ResNet-50	0.998	0.881	1.000	0.868	1.000	1.000

• Batch size=64, activation=ELU:

  Dataset	Model	ZLG ClsAcc	ZLG InsAcc	LLG ClsAcc	LLG InsAcc	Our ClsAcc	Our InsAcc
  MNIST	LeNet	1.000	0.948	0.580	0.283	0.990	0.969
  CIFAR-10	ResNet-18	1.000	0.902	0.980	0.697	1.000	0.972
  CIFAR-100	ResNet-50	0.985	0.897	0.845	0.603	1.000	1.000

Q9: Can the authors test their methods in a FedAvg setting (check [3,6,7])?

We present a concept for a label attack within the Federated Averaging (FedAvg) setting. Drawing inspiration from [7], we leverage a surrogate model to estimate the unshared gradients, denoted as $G_t$, during local training with FedAvg. At each iteration $t$, we update the model $M_t$ by considering the previous estimated state $M_{t-1}$, the update $G_{t-1}$, and the learning rate $\eta$. This update is represented by the equation $M_{t} = M_{t-1} - \eta \cdot G_{t-1}$. Utilizing the updated model $M_{t}$ and the estimated gradients $G_{t}$, we can calculate positive ($\hat p_j^+$) and negative ($\hat p_j^-$) probabilities for each class $j$. These probabilities are derived through our formulated expression for FedSGD algorithm. At this point, we can restore all the labels during the local updates.

[7] Zhu, Junyi, Ruicong Yao, and Matthew B. Blaschko. "Surrogate model extension (SME): A fast and accurate weight update attack on federated learning." arXiv preprint arXiv:2306.00127 (2023).

Q10: Do the authors know why the SELU activation results in so much more variance in the estimates of its posterior probability distribution?

We address this phenomenon by examining the output range $a$ of various activation functions. The input is denoted as $x$, and we all use the default settings in PyTorch.

• Sigmoid: $a\in(0,1)$
• Tanh: $a\in(-1,1)$
• ReLU: $a\in[0, x]$
• ELU: $a\in(-1,x]$
• SELU: $a\in(-1.76,x]$

Observing these output ranges, it becomes evident that SELU has the widest output space among these activation functions. Consequently, the output of each layer in the network may undergo larger shifts compared to the others. This increased shift results in a posterior probability distribution with higher variance, posing a greater challenge for attackers attempting estimation. Understanding this aspect sheds could explain why the attacks under SELU activation have a relative worse performance compared to the other activation functions.

Q4: Can the authors expand the focal loss experiments to more settings, especially ones where the success rate is not 100%?

Here, we present additional experiments conducted on the Focal Loss. We mainly test the parameters of $\tau$, $\gamma$ and $\varepsilon$ on an untrained model, and average the experiments over 10 trials. Focusing on temperature $\tau$, we present several cases where the accuracy is not 100%. In addition, by varying $\gamma$ and $\varepsilon$ on these setting, the accuracies are not affected.

Temperature $\tau$ (batch size=64, activation=ReLU):

We have observed that the temperature parameter, $\tau$, significantly impacts smaller datasets. When $\tau$ is smaller, the space of logits expands, complicating the estimation of batch posterior probabilities. Consequently, as $\tau$ decreases, label accuracy deteriorates. For the large datasets with more classes, such as CIFAR-10, the logits space is hardly influenced by changing different $\tau$.

Q5: Can the authors provide a discussion about why the focal loss is significant in particular? Are there other common classification losses that it covers as sub-cases? Are there other common classification losses not covered by the focal loss that might be more secure?

• To the best of our knowledge, Focal Loss has the general form in the cross-entropy loss variants and it can be converted to CE loss or BCE loss by setting different $\alpha$ and $\gamma$. We aim to derive a general form of label leakage from gradient, so we choose the Focal Loss.
• In supervised learning, we currently did not find other common classification losses that can cover the Focal Loss.
• There are indeed some other classification losses not covered by the Focal Loss, such as the the Hinge Loss used in Support Vector Machines or the Kullback-Leibler Divergence used in some probabilistic models. However, this paper cannot conclude they are more or less secure than the Focal Loss. Because this paper only covers the classification loss function with cross-entropy and softmax (or sigmoid).

Note that neither me, nor the other reviewers seem to see your updated paper. Can the authors upload it?
Edit: Now seem to be ok. Thank you.

We greatly appreciate your meticulous reading this paper and providing so many valuable suggestions. Regardless of the final result, we are very lucky to have a reviewer like you! We carefully read your comments and rethink about how to improve this work. We will try our best to answer your questions and concerns. We are pleased to offer more explanation or revisions if you have any other recommendations.

Q1: Can the authors fix consistently throughout the paper and appendix the definition and usage of the focal loss?

In Appendix A.1, we step-by-step derive the Focal Loss in multi-class scenarios from the original binary Focal Loss in [1].

Some important points we would like to emphasize:

• Difference in probabilities: $p_i$ is the post-softmax probability, while $p_t$ represents the automatic determined confidence of the input at class $i$, where $t=i$.
• Mechanism of weight: For an easy-to-learn sample, $p_t$ might be close to the target label. So, Focal Loss assigns a small coefficient $(1-p_t)^{\gamma}$ as the weight. However, for a hard-to-learn sample, $p_t$ may be close to 0. Then $(1-p_t)^{\gamma}$ a relative large weight to enhance the ratio of the these samples in the total loss.
• Summation sign: Since the binary Focal Loss [1] just has one output, the summation is not necessary. In the multi-class case, we use the summation to cover all the classes $t\in[1,K]$, and aim to derive a general conclusion in Theorem 1.
• Why Focal Loss: To the best of our knowledge, Focal Loss has the general form in the cross-entropy loss variants and it can be converted to CE loss or BCE loss by setting different $\alpha$ and $\gamma$. We aim to derive a general form of label leakage from gradient, so we choose the Focal Loss.

[1] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollar. Focal loss for dense object detection. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2980–2988, 2017.

Q2: Can the authors provide an extension of the current theoretical analysis to the general exponential family or at least to the subset of the exponential family covering the focal loss?

We would like to explain the reason of introducing the Exponential Family. Actually, we introduce the Exponential Family to better understand why the combination of cross-entropy and normalization functions (softmax / sigmoid) leads to the conclusions of Theorem 1 in this paper. Although the studies like iDLG have deduced some conclusions about cross-entropy and softmax, they do not delve into the underlying reasons. For illustration, we also find that sigmoid + cross-entropy also satisfies this conclusion. We also wonder if there are other functions that can replace softmax and whether the label leakage still holds? So we ask ourselves the following three questions to:

1. How did softmax (or sigmoid) originate and why is it applied to classification problems?
2. Why can cross-entropy serve as a loss function for classification problems?
3. When softmax and cross-entropy are combined, why does the term $(\boldsymbol p-\boldsymbol y)$ appear?

The exponential family can provide satisfactory answers to these questions. The Focal Loss is a variant of the common CE loss, and its unique term $(1-p_t)^{\gamma}$ represents the auto-determined weight for class imbalance. We find that from the perspective of exponential family, the cross-entropy loss and softmax (or sigmoid) are naturally combined with each other. To this end, we make sure that no other functions can replace softmax (or sigmoid), and the occurence of label leakage is in evitable in a classification model shared gradients.

The Exponential Family also encompasses the Bernoulli, Poisson and Gaussian distributions. However, these distributions do not support the conclusion proposed in this paper. Therefore, the conclusion is only applicable to the Categorical distribution within the Exponential Family.

Q3: Can the authors expand the discussion about the interpretation of their theoretical findings, in particular explaining in more detail their computational and privacy arguments in the context of the full exponential family?

In Appendix A.4, we expand the derivation of the theorectical findings. Please refer to that part.

I really appreciate Appendix B. It really helps me to evaluate the paper's contribution better.

For the next revision, I will suggest authors provide a version of Table 5-7 including the main results -e.g iLRG and the version of LLG/ZLG that does not use auxiliary data (but dummy data), as I think it will benefit the community as a whole, as these works have been inconsistent at comparing and citing related work. I also think trying the author's approach on dummy data might be interesting.

For the next revision, I will also encourage the authors to decrease the prominence of the exponential family discussion in the main paper and/or to extend it to the focal loss.

I am very thankful to the authors for their rebuttal engagement as a whole, and I have raised my grade to 6. I might raise it further after going through the paper fully with the new information supplied. I could not do this in the short time frame given.

Thanks for appreciation of our theorectical analysis and empirical evaluations, and pointing out the imperceptible mistakes. We give a point-by-point response to the weaknesses and questions, and hope to clear up your concerns.

Weakness 1: The proposed method requires an auxiliary dataset with the same distribution of training data. This requirement is impractical in FL.

We would like to highlight the following points to justify this assumption:

1. Realistic Threat Model: In many practical scenarios, attackers often have access to auxiliary data sources that closely resemble the distribution of the target training data. This assumption aligns with the notion that attackers may gather information from publicly available sources or other related datasets to enhance the effectiveness of their attacks.
2. Common Practice in Security Analysis: Making use of auxiliary data with a similar distribution is a standard practice in security analysis, especially in the research fields of Membership Inference Attacks [1, 2] and Gradient Inversion Attacks [3, 4]. This enables a more thorough evaluation of the model's robustness, covering a broader range of potential adversary scenarios.
3. Worst-Case Scenario Exploration: Assuming auxiliary data with a matching distribution, our analysis explores a worst-case scenario, where the adversary accesses highly relevant information. This reveals insights into the FL's vulnerabilities under challenging conditions, providing a comprehensive evaluation of its security properties.

[1] Hu, Hongsheng, et al. "Membership inference attacks on machine learning: A survey." ACM Computing Surveys (CSUR) 54.11s (2022): 1-37.

[2] Carlini, Nicholas, et al. "Membership inference attacks from first principles." 2022 IEEE Symposium on Security and Privacy (SP). IEEE, 2022.

[3] Jeon, Jinwoo, et al. "Gradient inversion with generative image prior." Advances in neural information processing systems 34 (2021): 29898-29908.

[4] Balunovic, Mislav, et al. "Lamp: Extracting text from gradients with language model priors." Advances in Neural Information Processing Systems 35 (2022): 7641-7654.

Weakness 2: I cannot find the exact definition of negative probabilities and positive probabilities in Sec 5.1, class-wise labels in Theorem 2.

Here, we add the definition and explanation of the positive and negative probabilities, and the class-wise labels.

In a multi-class classification problem, each instance in the dataset belongs to one of several classes. Let's denote the set of classes as $K$ and a particular class of interest as $k\in K$. In this context, we can define positive and negative samples for class $k$.

1. Positive Samples ($X_{k}^{+}$): The positive samples of class $k$ satisfy that: $X_{k}^{+}=\{x_i|y_i=k\}$, where $x_i$ is the input and $y_i$ is the corresponding label.
2. Negative Samples ($X_{k}^{-}$): Similarly, the negative samples of class $k$ satisfy that: $X_{k}^{-}=\{x_i|y_i\neq k\}$.

According to the positive and negative samples, we can then get the positive and negative probability for class $k$.

1. Positive Probability ($p_{k}^{\text{+}}$): When a positive instance is fed into the model, the predicted probability of class $k$ is termed the positive probability. Since the Softmax activation function is used in the output layer, the output posterior probability $p^{+}$ is a vector of length $k$. Therefore, the positive probability for class $k$ can be expressed as $p_{k}^{+}$.
2. Negative Probability ($p_{k}^{-}$): Similarly, when a negative sample is input into the model, the $k$th element of the output probability vector represents the negative probability, denoted as $p_{k}^{-}$. It's essential to note that any negative sample associated with the other $(K-1)$ classes contributes to $p_{k}^{-}$.

When using an auxiliary dataset to esimate the probabilities of the target training batch in FL, we denote the estimated positive and negative probabilities as $\hat p_{k}^{+}$ and $\hat p_{k}^{-}$, respectively.

In a batch size of $B$, we aim to recover the labels of each instance within the batch, i.e., $\mathbf{y}=[y^{(1)},y^{(2)},\cdots,y^{(B)}]$. As this is a multi-class classification problem, the ground-truth labels $\mathbf{y}$ can also be represented the occurences of each class: $\mathbf{y}=[n_1,n_2,\cdots,n_K]$, where $n_k$ is the number of samples belonging to class $k$ and $K$ is the number of total classes.

• Class-wise Labels: In a batch data, the class-wise labels can be defined as: $n_k=\sum_{i=1}^{B}\delta(y^{(i)}=k)$. Here, $n_k$ is the number of samples belonging to class $k$, B is the batch size, $y^{(i)}$ is the true class label of the $i$th instance in the batch, and $\delta(\cdot)$ is the Kronecker delta function, which equals 1 if the condition inside is true and 0 otherwise.

We appreciate your praise for the novelty and inspiration of our paper and some concerns about the assumption and experiments. We give point-by-point responses below. We are pleased to offer more explanation or revisions if you have any other recommendations.

Weakness 1: The attack assumes adversary can use an auxiliary data with the same distribution of training data, which in reality is not always true.

We would like to highlight the following points to justify this assumption:

1. Realistic Threat Model: In many practical scenarios, attackers often have access to auxiliary data sources that closely resemble the distribution of the target training data. This assumption aligns with the notion that attackers may gather information from publicly available sources or other related datasets to enhance the effectiveness of their attacks.
2. Common Practice in Security Analysis: Making use of auxiliary data with a similar distribution is a standard practice in security analysis, especially in the research fields of Membership Inference Attacks [1] and Gradient Inversion Attacks [2]. This enables a more thorough evaluation of the model's robustness, covering a broader range of potential adversary scenarios.
3. Worst-Case Scenario Exploration: Assuming auxiliary data with a matching distribution, our analysis explores a worst-case scenario, where the adversary accesses highly relevant information. This reveals insights into the FL's vulnerabilities under challenging conditions, providing a comprehensive evaluation of its security properties.

[1] Hu, Hongsheng, et al. "Membership inference attacks on machine learning: A survey." ACM Computing Surveys (CSUR) 54.11s (2022): 1-37.

[2] Jeon, Jinwoo, et al. "Gradient inversion with generative image prior." Advances in neural information processing systems 34 (2021): 29898-29908.

Weakness 2: I hope there can be some more discussions on how the results can be if auxiliary datasets and training datasets have considerable distribution shift.

To address concerns about significant distribution shifts, we conducted supplementary experiments, averaging results over 10 trials.

1. In the first set of experiments, we alternately used CIFAR-10 and CINIC-10 [3] as the training and auxiliary datasets. CINIC-10 extends CIFAR-10 by incorporating downsampled ImageNet images. While there is some overlap in the distributions due to similar object categories, there is a notable bias in the data features. We employed an untrained VGG model with a batch size of 64, and the results are presented in the table below.

   Training Data	Auxiliary Data	Model	Our ClsAcc	Our InsAcc
   CIFAR-10	CINIC-10	VGG	1.000	1.000
   CINIC-10	CIFAR-10	VGG	1.000	1.000

2. Similarly, the second set of experiments involves alternating between MNIST and Fashion-MNIST as the training and auxiliary datasets. Despite both datasets having 10 classes, the objects they represent are entirely different. MNIST dataset contains a lot of handwritten digits, while Fashion-MNIST represents the article of clothing. Employing an untrained LeNet model with a batch size of 64, the results are presented in the table below.

   Training Data	Auxiliary Data	Model	Our ClsAcc	Our InsAcc
   MNIST	Fashion-MNIST	LeNet	1.000	0.969
   Fashion-MNIST	MNIST	LeNet	1.000	0.948

This phenomenon can be explained as follows: In the initial phase of model training, the model lacks the ability to differentiate between samples from each class, assigning similar output probabilities to all fitted samples (i.e., $1/K$). For different training datasets like Fashion-MNIST and MNIST, the model projects input figures to similar output probabilities with slight variations. This benefits our label recovery attack as it becomes easier to estimate the posterior probabilities of the target batch.

Given that Fashion-MNIST is more intricate than MNIST, utilizing MNIST as the auxiliary dataset poses challenges in accurately estimating the probability distribution of batch training samples. This difficulty accounts for the marginal decrease in InsAcc. However, since CIFAR-10 and CINIC-10 exhibit similarities, there is no difference in InsAcc. The outcomes of these experiments illustrate that if an attacker initiates an attack during the early training stages, having an auxiliary dataset with an identical distribution to the training dataset may not be essential.

[3] Darlow, Luke N., et al. "Cinic-10 is not imagenet or cifar-10." arXiv preprint arXiv:1810.03505 (2018).