Hiding in Plain Sight: Disguising Data Stealing Attacks in Federated Learning

\newcommand{Rfive}{\textcolor{purple}{ZL2M}} Q5.8: Table 3 is counterintuitive—can you provide an explanation and more exploration?

As more exploration we offer new experiments on distribution shifts with medical data (GQ4) which support our prior conclusions. We have failed to understand what in Table 3 the reviewer found counterintuitive, and would gladly discuss further if the reviewer could elaborate.

Q5.9: Does the property always exist for each sample in the batch; how does this affect SEER?

Interesting question. Rigorously speaking, yes—for each image $x$, the property “equal to $x$” will single out only this image. So the actually interesting question is can each image be singled out by a property that is (i) simple enough to well train malicious weights (ii) transferable to other batches without retraining, as this is one of the key features of SEER (see Q4.6). As we do not target particular images but simply aim to demonstrate that some of the user’s private images can be leaked (a standard understanding of privacy; see Q5.6), this question has no immediate impact on SEER. Whatever property we chose in our experiments, a once-trained SEER model was able to recover images from many batches with high success (App. C.6).

Q5.10: Can you compare with more MS attacks such as Zhang23 and Zhao23 (LOKI)?

As we note in the paper, these and similar works either focus on much stronger threat models or are easily detectable, thus we believe it is not appropriate to include them in Sec. 5. Regardless, prompted by this question, we ran two new experiments following the setup of Table 4, obtaining expected results.

Recall that in Table 4 SEER obtains UndRec% of 90 (in 90% the attack is undetectable, as we always have DSNR<5 and the transmission metric<0.3) and also obtains good reconstructions.

Zhang23: As noted in Sec. 3, this is a boosted analytical attack, and thus has the following two drawbacks:

1. Their transmission (passing inputs unchanged to the first FC layer; necessary for convnets) does not scale to realistic networks such as ResNet due to stride>1 and pooling—accordingly, these results are not present in the cited work. As we cite in our paper, Fowl et al., 2022b show that transmitting this way up to the final pooling already severely downscales the image. We further find that past that pooling (thus pre-first FC), the images can not be transmitted with any success.
2. Ignoring the scalability issue, the transmission used in this work is easily weight-space detectable on any convnet (see App. A).

Despite these, we ran Zhang23 in the setting of Table 4, taking max PSNR of batch reconstructions. The results follow:

Rec%: 5 | UndRec%: 0 | PSNR: 15.8 +- 1.8 | PSNR-und: N/A | PSNR-und-rec: N/A

As expected, while inputs to the first FC layer were recovered well (average PSNR of 43), this almost never leads to an image reconstruction, i.e., Rec% is only 5. Moreover, the transmission metric (App. A) is $\infty$ in all cases (as opposed to <0.3 for natural networks and SEER), and thus, UndRec% is 0, and the PSNR of undetected reconstructions is undefined.

Zhao23 (LOKI) Similarly, as we note in the paper, LOKI focuses on a different (significantly stronger) threat model where architectural changes and sending different updates to clients (model inconsistency) are allowed. LOKI focuses on scaling attacks in this setting and succeeds in this goal. However it can, by design, never be directly applied to fixed/known architectures such as ResNet or in cases where clients get the same model.

To recall our Sec. 3, our main focus are practical cases with threat-aware clients, so we require use of known architectures (as clients can easily detect modules that are never used in practice, such as LOKI’s 1Conv+2FC at the start of the network) and no model inconsistency (as this is easily detectable and revertable with reverse secure aggregation [Pasquini et al., 2022]).

Due to the stronger threat model, LOKI can avoid the drawback (1) of Zhang23 stated above, but the issue (2) still stands as LOKI is fundamentally a boosted analytical attack, thus it can be detected. We confirm this experimentally on an augmented ResNet with 1Conv+2FC in front (as LOKI can not be applied in the exact setting of Table 4):

Rec%: 100 | UndRec%: 0 | PSNR: 143.4 +- 10.3 | PSNR-und: N/A | PSNR-und-rec: N/A

As expected, adding an FC layer to the front allows almost exact reconstruction, a known fact in the field [Phong et al., 2018; Geiping et al., 2020]. However, transmission is still needed, thus UndRec%=0, i.e. no images are reconstructed undetectably. Additionally, any threat-aware client could notice FC layers in front of the convolutional stack, which never occurs naturally.

We hope these results help the reviewer acknowledge our points about these attacks.

\newcommand{Rfive}{\textcolor{purple}{ZL2M}} We thank the reviewer $\Rfive$ for their extensive feedback; our responses are below. We will incorporate the clarifications and explanations below in future revisions of our paper. We are happy to continue the discussion in case more questions arise. GQ# refers to points in our general response.

Q5.1: Can you include the auxiliary data requirement in the threat model and provide proofs?

We acknowledge that this requirement could be already mentioned in Sec. 3 and will include this in the next revision. Our ablations with aux. data (see Q5.4) show that this is not restrictive. This contrasts with other aspects mentioned in Sec. 3, such as architecture modifications, which are trivially detectable yet still considered viable by many prior works. We believe our threat model is much more restrictive for the attacker and more realistic, and the success of SEER despite this hardening, is our key contribution.

We are not sure what the reviewer refers to as proofs for the threat model—if the reviewer can elaborate, we are happy to discuss further.

Q5.2: What client-side checks does the attacker need to evade, and how do you show that SEER is undetectable?

We argue in the paper that the attacker should evade all checks known to practitioners at that point in time (a standard security setup). This was ignored in prior work, inflating the perceived danger from those attacks, which we show are easily detectable. In our paper, SEER is benchmarked and is not detected by D-SNR (Figure 1, Table 4) and the transmission weight-space detection (App. A)---two principled checks that we show can detect all prior work in our threat model. Importantly (see Outlook), we do not claim that no future detection method will be able to detect SEER, and we encourage work on complex yet principled detection methods.

Q5.3: What is the meaning of “handcrafted modifications” and why is SEER not in this group?

As “handcrafted modifications” we dub manual changes of the model, such as setting weights to 0 or 1 “by hand”, as these often directly imply trivial detectability—we explain the reasons in the paper. SEER only changes weights via continuous optimization, thus has no handcrafted changes—while this makes SEER the only attack not detectable at the moment, as we mention in Q5.2 and the paper, it is possible that future work changes this, but we see no easy path there. We do agree that “handcrafted modifications” is not a rigorous mathematical definition, but we still find it useful as a means to accurately discuss prior work.

Q5.4: Can you clarify how SEER breaks secure aggregation?

On a high level, breaking secure aggregation reduces to disaggregating individual gradients from their aggregated sum/mean and is thus not fundamentally different from attacks on large batches when no batchnorm (BN) layers are present in the network (See Q2.3 for what happens under BN). Disaggregation of individual sample gradients from their sum is possible in the MS setting in practice, as was shown in prior work, e.g., [Fowl et al., 2022b]. Our contribution is an attack that is not easily detectable in gradient and weight space (See Sec. 3). We provide further technical details on the variant of SEER we use for secure aggregation in App. B.

Q5.5: Can you provide more ablation studies on the size and type of auxiliary data?

Beyond Table 3, prompted by this question we show results of two new experiments: in GQ4 we investigate a strong distribution shift where clients use medical data, and in GQ5 we ablate the size of the aux. corpus. In both cases, we find that the aux. data assumption of SEER is not restrictive.

Q5.6: Is it necessary to train a separate SEER model for many properties?

No. A SEER model trained once can be applied across different clients, training rounds, and even FL deployments and will, whenever successful (88-98% of the time in the settings of our experiments), produce high-quality reconstructions of private user images that satisfy the property. We argue, in sync with prior work, that privacy is an absolute property—as long as any private client image is leaked to attackers, this is a privacy leak and breaks the key promise of FL. A good analogy is security, where a similar perspective is widely accepted.

Q5.7: Can you evaluate SEER under defenses?

Certainly, we agree that this is important. See GQ3, where we, prompted by this question, perform a new experiment and show that SEER still succeeds under differential privacy. Additionally, we point the author to Figure 1, Figure 4 and Appendix A, where we evaluate SEER under two principled client-side checks that we developed and confirm they are very effective as defenses against most prior work—yet, they do not break SEER.

\newcommand{Rfive}{\textcolor{purple}{ZL2M}} We thank the reviewer $\Rfive$ for engaging with the rebuttal and following up on Q5.9. We elaborate on our answer above, and explore some related questions:

(1) For a given d-dim example $x_i$ in a batch of size B, is there always a property that separates it from the rest of the batch?
Above we gave a positive answer in the general case. Here, we note that the answer is still positive even for restricted properties $m(x) < \tau$, where $m$ is a linear measurement, which we introduce on top of Sec 4.1 and use in SEER. The underlying question here is if the hyperplane $m(x)-\tau=0$ can always separate the point $x_i$ from the rest. This is answered positively by a well-known result from theoretical ML (see e.g., [1]) which states that the VC dimension of affine classifiers (in this case $sgn(m(x)-\tau)$) in $\mathbb{R}^d$ is $d+1$. Thus, such a property always exists when our images are in a general position (which usually holds) and $B \leq d+1$ (which holds even for low-dim CIFAR10, as $d+1=3073$ which is far above practical batch sizes).

(2) Can such a property be efficiently constructed?
Yes. This is equivalent to finding any linear hard-margin SVM on linearly separable data, while ignoring the constraint of margin maximization—this can be encoded and exactly solved as a simple linear program (LP) in polynomial time of the input size $d$.

(3) How does this look like in practice with SEER?
In practice, we obviously do not know the images we are trying to reconstruct and do not target any particular image but demonstrate unexpected privacy leakage in FL by showing that some client data can be always leaked in practical deployments. The question of which properties are easier/harder to train SEER with is interesting and we offer an initial exploration in App. C.6 where we test many properties (Bright/Dark, Red/Green/Blue, Hor/Ver Edges, combinations of previous, random properties) and show that SEER can be trained well with all of them to reconstruct substantially different types of images (see Fig. 5).

We hope this clarifies Q5.9. We encourage the reviewer to follow up on any of Q5.1-Q5.10 if there are still outstanding questions, and reassess our paper in light of our answers.

[1] https://mlweb.loria.fr/book/en/VCdimhyperplane.html

\newcommand{Rfour}{\textcolor{green}{sMvX}} We are honored that the reviewer $\Rfour$ found both our proposed attack and defense techniques innovative and intriguing. We appreciate the raised questions which prompted us to conduct additional experiments, shown in our general answer. Below we provide pointers to those experiments. We are happy to continue the discussion in case of additional questions. Note that GQ# refers to points in our general response.

Q4.1: Can you provide additional distribution shift experiments where clients have specialized (e.g., medical) data, and SEER is trained on a generic dataset (CIFAR10)?

In GQ4, we report the results of exactly the experiment described by the reviewer on the ISIC2019 dataset. As on this dataset, our method performs even better than on TinyImageNet, we remain confident that SEER is robust regarding the nature of the client data.

Q4.2: Can differential privacy defend against SEER?

We agree that this is an important aspect to evaluate. See our new experiment in GQ3 prompted by this question, where we demonstrate that DP is unfortunately not an effective defense.

Q4.3: How does SEER perform in non-IID settings?

See our new experiment in GQ2 prompted by this question, where we notice no significant degradation in the non-IID case.

\newcommand{Rthree}{\textcolor{red}{SRu7}} We thank the reviewer $\Rthree$ for their overall positive view of our work. We are pleased that they found our defense strategies principled and important and appreciated our discussion of the impact of our work on privacy and fairness of FL. We answer outstanding questions below. We are happy to continue the discussion in case of additional questions. Note that GQ# refers to points in our general response.

Q3.1: Can you outline possible mitigation strategies?

Please see our new experiment in GQ3, where we show that commonly used differential privacy is not an effective defense against SEER. Further, as we point out in the paper, we find the restriction of protocols to disallow obvious malicious modifications, such as changing the architecture or sending different updates to clients, to be the most important precaution to take against privacy attacks in the MS setting. Beyond this, we find principled client-side detection to still be the most promising direction (as also discussed in Q2.2). While none of the currently existing defenses work against SEER, we hope that our paper further emphasizes the need for research on detection and are looking forward to new results in this area.

We are delighted the reviewer $\textcolor{blue}{dXt6}$ identified our method as addressing many shortcomings in prior state-of-the-art methods and considers our methodology novel. We are thankful for the reviewer’s insightful questions. We provide exhaustive clarifications on SEER’s technical details, as well as pointers to the additional experiments conducted for the reviewer below. We are happy to continue the discussion in case of follow-ups. GQ# refers to our general response.

Q2.1: Are the need for offline training and auxiliary dataset limitations of SEER?

We believe not. To confirm this, we show that SEER is robust to distribution shifts between client and aux. data (Table 3 and a new experiment in GQ4), and that it does not require a large aux. corpus (a new experiment in GQ5). Offline training is not a limitation as it is independent of the FL deployment (as we show knowledge of the private data distribution is not needed) and can be thus done at any time before the main FL training starts.

Q2.2: Can SEER be detected by observing the loss over rounds or comparing its weights to benign model weights?

SEER does not require application in more than one round, and such attacks are often applied in the first round [Balunovic et al., 2022b], as the reviewer suggests. Further, it is known that in FL, per-client loss values can be noisy, even if the global loss consistently falls. As noted in [Boenisch et al., 2021], this makes loss-based detection unlikely to work. Regarding weights, as SEER only modifies them through continuous optimization, it uses no obvious handcrafted patterns that can be flagged (as opposed to prior work analyzed in the paper). However, as we note in our Outlook section, we do not claim that future refinements of this idea will fail to mitigate SEER, and find this a good direction for future work. As SEER is resistant to all current defenses (see also GQ3 for our experiment on DP), we believe our work is an important contribution and can motivate future defenses.

Q2.3: How are BN statistics used to pick the local property? When this is impossible, can we use linear search?

Unlike prior work, we do not rely on BN statistics, but show that the mere presence of BN harms privacy. Consider the no-BN case, where the final gradient is a mean of sample gradients, and the forward passes (and thus the gradients) of samples are independent. When BN is used, the final gradient is still a simple mean, but the computation of sample gradients is intertwined, i.e., $g(x_i)$ and $g(x_j)$ depend on each other. This is an issue because it enables a MS attacker to influence $g$ to make e.g., the gradient of $x_i$ non-negligible iff $x_i$ is the brightest image in the batch, disaggregating this example. More formally, BN breaks the implicit independence assumption in the calculation of $p(X=1)$ in the proof of Proposition 3.2 in [Wen et al., 2022], allowing to improve on its attack success upper bound. SEER does exactly this, achieving empirically much better success rates than bound’s $\frac{1}{e}$ by training weights to be attackable, incentivizing disaggregation through our loss.

This is impossible without BN as the computational graph of $x_i$ is independent of other samples, thus there is no way to find the “brightest image”. Finding the optimal threshold of the property that the reviewer asks about is independent of this—even with the optimal property, if there is no BN, we cannot get attack success above the bound from [Wen et al., 2022], unlike SEER in the BN case.

Q2.4: Can we replace the learned disaggregation with handcrafted criteria?

We believe this is not possible, as designing an undetectable disaggregation scheme is inherently hard. In particular, we distinguish between (1) choosing a property metric $m$ (e.g., Most Bright), which in SEER we do manually (see App. C.6 where SEER is shown to be robust to property choice) and where the sample loss could be used as a criterion, and (2) finding malicious weights that isolate the image with that property. Point (2) is very hard without trivially detectable changes, which is the very motivation for SEER, and what we see as our main contribution.

Q2.5: How does SEER perform in the case of non-IID client data?

We do not notice significant degradation in this case—see our new experiments in GQ2.

Q2.6: Can you outline possible mitigation strategies besides DP/HE?

As we point out in the paper, we find the restriction of protocols to disallow obvious modifications, such as changing the architecture or sending different updates to clients, to be the most important precaution against privacy attacks. Beyond this, we find principled client-side checks to be the most promising direction (as discussed in Q2.2, Outlook section). While no existing defenses work against SEER, we hope that our paper emphasizes the need for research on detection and are looking forward to new results in this area

\newcommand{Rone}{\textcolor{orange}{Lg7w}} We are glad the reviewer $\Rone$ found our proposed metric D-SNR useful for detecting existing MS attacks and found our newly proposed attack SEER effective. We address the raised concerns below. We are happy to continue the discussion in case of additional questions. Note that GQ# refers to points in our general response.

Q1.1: Does the performance of SEER depend on the setting?

Certainly—some settings can be inherently harder for the attacker, e.g., in our Table 1 SEER’s accuracy is always a few % lower in the CIFAR10+Bright than in the CIFAR100+Red setting. However, in both of these settings as well as in our experiments on ResImageNet and ImageNet (see GQ1), and across various properties and batch sizes, SEER consistently succeeds in stealing user data. Thus, despite our considerable efforts in making the evaluation thorough, we are yet unable to find a setting where SEER underperforms.

Q1.2: When and why is it possible to disaggregate individual sample gradients from their mean in the MS setting?

The typical intuition about the disaggregation of a single element from a sum or a mean (in our case of gradients) is that it is not possible. In federated learning, this is formalized in the notion of Secure Aggregation, which provably secures the privacy of individual sample gradients in the honest-but-curious server setting. However, Secure Aggregation can be attacked in the MS server setting. This is due to breaking a key assumption in the theory that the attacker has no influence over the gradient function $g(x)$. In the MS setting, $g(x)$ can be influenced partially by choosing the global model weights $\theta_f$ sent to the clients. Prior work has leveraged this observation in practice (e.g. See [Wen et al., 2022]) to disaggregate gradients from their mean almost independently of the client data with very good results. Unfortunately, such work, as discussed in Section 3, is trivially client-detectable. We see the results of SEER across several different properties (App. C.6), and datasets (Section 5) to be a strong indication that the same strong disaggregation results are also possible when the attack is not easily detectable.

Q1.3: Can you show results on more datasets beyond CIFAR10/CIFAR100?

We refer the reviewer to our results on ResImageNet that were included in our original submission as a way to investigate high-resolution settings (top of page 8), as well as our new results on ImageNet that were prompted by this reviewer’s question (GQ1), and our new distribution shift results on domain-specific medical data (GQ4).