Protecting Sensitive Data through Federated Co-Training

Dear reviewer UQKJ,

Thank you for your evaluation. We will address your questions point-by-point.

• The main motivation for our work is to improve privacy in federated learning. As stated in the abstract and introduction, federated learning promises to protect data privacy by only sharing model parameters, but in fact model parameters reveal a lot about training data. Differential privacy mechanisms can be used to improve privacy, but their privacy-utility trade-off is not ideal. Therefore, improving privacy is a very relevant problem. We approach this problem by first noting that in many application scenarios where privacy is paramount, e.g., healthcare, large public unlabeled datasets exist. Our approach uses such an unlabeled dataset to exchange information between clients. Distributed distillation and its variants have shown that by sharing soft labels for such an unlabeled datasets we can train neural networks collaboratively. We show that sharing soft labels indeed improves privacy over sharing model parameters, but does not fully protect private data. By sharing hard labels instead, we are able to protect privacy to the highest degree - at least in terms of existing membership inference attacks - while maintaining the model quality of sharing model parameters and soft labels. We hope that this clarifies our motivation. We will improve the description of our motivation in the manuscript.

• Model quality is canonically measured in terms of test accuracy as a proxy for the generalization error. Privacy is measured empirically by the likelihood of an attacker being able to conduct a successful membership inference attack (as described in Sec. 5 “Experimental Setup”, see also the probabilistic interpretation of ROC AUC), and it is determined theoretically via differential privacy (as described in Sec. 4).

• As discussed in the first point, our motivation is to improve privacy while maintaining model quality in federated learning. We do so by using an unlabelled dataset.

• Our insights are that sharing hard labels instead of model parameters or soft labels indeed improves privacy - what was to be expected - and surprisingly does not decrease model quality. Moreover, we find that sharing hard labels has the additional benefit of allowing us to train interpretable models, like decision trees and rule ensembles, in a federated way. We can even train ensembles of various local models, as we show in the general comment.

• Our Section 3 explains the scenario, introduces the novel method FedCT and analyzes its convergence and communication complexity. Since privacy is the main motivation for our work, we show in Section 4 how a privacy guarantee based on differential privacy can be derived for FedCT. For this, we provide a novel sensitivity analysis (Proposition 2), which is required to provide a meaningful privacy guarantee.

Thanks for the response.

Dear reviewer W2md,

We appreciate your detailed assessment of our work and we are happy that you find the problem relevant and our theoretical privacy analysis and the employed XOR mechanism interesting. We address your questions point-by-point.

Weaknesses:

W1: We are aware of the wide range of semi-supervised federated learning methods, but to the best of our knowledge, FedCT is novel in that it shares hard labels on a public unlabeled dataset. This improves privacy substantially while maintaining model quality. Moreover, it allows us to train interpretable models. Since sharing hard labels hasn’t been explored in the literature, achieving differential privacy is not straight-forward. We employed the XOR mechanism for binary data, which required us to derive a novel bound on the sensitivity of classification algorithms. Regarding the papers you have referenced, [4] is not applicable since it assumes a public labeled dataset, [3] shares soft labels similar to distributed distillation, and [1],[2],[5],[6] share model parameters and therefore do not improve privacy over FedAvg at all.

W2: As our experiments show, the vulnerability of sharing soft labels to known membership inference attacks is lower than sharing model parameters, as one would expect, but still considerable. Since sharing hard labels achieves a model quality similar to sharing soft labels, its additional privacy is a clear advantage. Together with the fact that sharing hard labels allows us to train interpretable models, or mixed models (see the general comment), this means that FedCT has clear advantages over both sharing model parameters and soft labels.

W3: We used the term semi-honest as defined in [7] and used widely in cryptography and security, but we are happy to change the term to honest-but-curious in our manuscript.

W4: We have compared FedCT to model parameters sharing and soft label sharing in terms of accuracy and privacy, as well as the common differential privacy mechanism applied to federated averaging. Our empirical privacy evaluation measures the success in terms of ROC AUC of an attacker to perform state-of-the-art membership inference attacks on the information that is shared - model parameters, soft labels, or hard labels. Therefore, the resulting vulnerability scores are comparable. We do not see how we can attach our method of sharing hard labels to other federated semi-supervised learning methods that share soft labels or model parameters. Could you please elaborate how you envision such experiments?

W5: We did not extensively explore scenarios of too strong heterogeneity. Setting $\alpha$ to 2 yields the class distribution shown in Figure 2. The impact of strong heterogeneity on consensus is intricate and depends on factors such as the local learning algorithm, model, number of clients, and consensus mechanism robustness. Investigating this is very interesting, but it falls beyond the scope of this work.

Figure 2: Class distribution for $\alpha = 2$

Questions:

َQ1: Regarding the FedSSL methods you referenced, [1],[2],[5],[6] share model parameters and therefore do not improve privacy over FedAvg, [3] shares soft labels similar to distributed distillation, and [4] assumes a public labeled dataset and thus is not applicable to our scenario.

Q2: Since the information shared by clients in FedCT are binary matrices, we cannot apply the Gaussian or Laplacian mechanism, but need a noise mechanism tailored to binary data. The XOR mechanism was recently developed for binary data and provides a differential privacy guarantee, given that we can quantify the sensitivity. Since we cannot use clipping, like in the Gaussian mechanism for federated averaging, determining the sensitivity is non-trivial. In our Proposition 2, be derive a novel bound of the sensitivity for on-average-leave-one-out-stable learning algorithms, which includes a wide range of machine learning methods.

Q3: The reason why there are no baselines is simply that to the best of our knowledge, there are none. There exist several approaches that train decision trees in a semi-federated manner by computing the information gain in a distributed way (and protecting privacy via encryption)[8], but there is no method that trains local models collaboratively. The most informative baseline, thus, is centralized learning.

References:

[7] Goldreich, Oded. Foundations of cryptography: volume 2, basic applications. Cambridge university press, 2009.

[8] Truex, Stacey, et al. "A hybrid approach to privacy-preserving federated learning." Proceedings of the 12th ACM workshop on artificial intelligence and security. 2019.

Dear reviewer W2md,

Thank you for your response. We are glad that most of your questions have been clarified and we are happy to address your remaining concern. We agree that data heterogeneity is an important problem in federated learning. We find in our experiments that for label heterogeneity, label sharing (both soft, as in distillation, and our hard label sharing) requires that client predictions are not too far off, so that we can still achieve a meaningful consensus. In our experiments, we therefore designed local datasets to have a part that is fairly homogeneous (Dirichlet with large $\alpha$), to ensure that each client has at least a little information about each class. The other part is heterogeneous (Dirichlet with smaller $\alpha$). The same could in expectation be achieved with an intermediate Dirichlet parameter $\alpha$, but we wanted to make sure data is of the desired form. We have used $\alpha=100$ for the fairly homogeneous part, and $\alpha=2$ for the more heterogeneous part (see the experimental setup in Sec. 5 and Appendix C.3, and our visualization of the data distribution in Fig. 2 in our previous reply). Since in many standard FL papers, stronger heterogeneity is assumed, we performed an additional experiment for this rebuttal where we use $\alpha=0.01$ for the heterogeneous part. The results in the following table show that FedCT still performs on par with both FedAvg and DD.

The experiments in FedMD [1] on heterogeneous data are very interesting. They use a private labeled dataset and a public labeled dataset with related classes - in this case, CIFAR100 which contains 100 classes from 20 superclasses (bicylce, bus, and car are all vehicles). The task of local models is to predict the superclass, but client datasets only contain one class per superclass. Thereby, the datasets are heterogeneous wrt. the class labels, but homogeneous wrt. the superclasses. This also ensures that a meaningful consensus can be achieved. While we do not assume a labeled dataset as FedMD does, we evaluate FedCT in this scenario as well. We achieve a test accuracy of $ACC=0.5106$ which is comparable to the test accuracy of roughly $ACC=0.5$ that FedMD reports - note that this comparison is favoring FedMD, since FedMD uses supervised pre-training (transfer learning) on the labeled public dataset, which we do not. We will include these new results in the updated version of our manuscript.

Dear reviewer e9mW,

We appreciate your comprehensive assessment of our work and appreciate the favorable evaluation of the soundness and theoretical analysis of our approach. We will address your questions point-by-point.

W1. Thank you for the references. In our related work section, we decided to discuss only those federated semi-supervised learning methods that directly fit our scenario, due to space limitations. That is, distributed clients hold a private labeled dataset, have access to a public unlabeled dataset and aim at training models collaboratively without sharing either data nor model parameters. Regarding your references, [1] assumes a public labeled dataset, [2, 4] require clients to share model parameters with the server, and [2] in addition assumes that the unlabeled dataset is only accessible by the server, [3] uses co-regularization for personalized federated learning, so we naturally compare to the non-personalized variant distributed distillation, and we discussed [5] in our related work section. We apologize for not having sufficiently clarified this. We will highlight the problem statement in our manuscript and add a discussion of more distantly related federated semi-supervised learning methods (including the ones you mentioned) in the appendix.

W2. The motivation for our paper is to maximize privacy in a federated learning scenario, where clients have access to an unlabeled public dataset. Indeed, sharing soft labels preserves privacy to a larger extent than sharing model parameters, as our experiments show for the first time, yet they still reveal a lot about private data (VUL >$0.6$). As discussed in the introduction, we wanted to go one step further by sharing hard labels. Indeed, this substantially improves privacy protection, with an empirically optimal privacy (VUL $\approx 0.5$). Of course, this is only optimal wrt. known membership inference attacks, which is why we developed the differentially private variant of FedCT. What was surprising to us is that strong improvement in privacy comes at virtually no loss in accuracy. That is, we achieve a test accuracy comparable to FedAvg and Distributed Distillation, the latter sharing soft labels. We conjecture that in practice the advantage of sharing soft labels over hard labels in terms of model performance is negligible, yet the additional privacy risk is significant. Moreover, sharing hard labels allows us to train interpretable models as well, and even use different models at each client (please see our general comment for further details).

W3. We selected the most prominent baselines for each type of methods, i.e., vanilla FedAvg as a representative for model sharing in homogeneous data scenarios, DP-FedAvg as its private variant, and distributed distillation for sharing of soft-labels (or co-regularization). We additionally compared FedCT to PATE for this rebuttal, a model distillation method that can be adapted to our scenario. The results in Table 1 indicate that PATE is outperformed by FedCT and all other baselines in terms of test accuracy. As discussed above, we cannot compare to the approaches you referenced, since they are not applicable in our scenario.

W4. The appendix is available as a separate document in the supplementary material. For convenience, we also included a full document (paper and appendix) in the supplementary material. If you have any trouble accessing the supplementary material, please let us know and we will upload it to our anonymized repository.

Dear reviewer e9mW.

We are happy we could address most of your concerns. Regarding your concern W2, we will gladly clarify the distinction between hard and soft label sharing in the manuscript. To reiterate: As mentioned in the introduction, sharing hard labels instead of soft labels improves privacy and allows us to train interpretable models. Since hard labels carry less information than soft labels, this could in principle lead to a decrease in model quality. In our empirical evaluation (particularly Table 1), we show that we cannot observe any significant decrease in model quality - both soft and hard label sharing perform comparable to model parameters sharing (FedAvg). Note that we use distributed distillation as the most prominent representative of soft label sharing. At the same time, our experiments support the claim that sharing hard labels improves privacy: soft label sharing has an average vulnerability of 0.61, whereas sharing hard labels has a substantially reduced vulnerability of 0.52. We summarize these differences in the following table. We will include a polished version of the table together with a paragraph explaining these details in the appendix.

Note that both soft and hard label sharing requires an unlabeled public dataset.

Dear reviewer NCnb,

Thank you for your detailed evaluation of our work, in particular that you highlighted the importance of the problem we tackle, our theoretical analysis, and privacy evaluation. We will address your questions point by point.

Questions:

(1) PATE is essentially a distillation algorithm with the goal to produce a single student model from an ensemble of teachers. To protect the private dataset the teachers have been trained upon, a Laplace mechanism is applied to their predictions - more precisely, their counts - before determining the consensus label. Thereby, a curious student cannot infer upon the private training data. In both PATE and FedCT, hard labels are used to form a consensus. The fundamental differences are that (i) PATE is not a collaborative training algorithm and that (ii) PATE therefore is not concerned with protecting against an honest-but-curious server. Regarding point (i): In a nutshell, PATE trains all teachers until convergence once, uses their predictions to pseudo-label the public unlabeled data, and then train the student on this pseudo-labeled dataset. Thus, teachers never improve. This fundamentally differs from FedCT where clients train models collaboratively by iteratively sharing hard labels to improve the pseudo-labeled dataset, which in turn improves local models, which again improves the pseudo-labeling. This difference also explains point (ii): In PATE, the student is separated from the teachers, so that the main privacy concern is protecting against a curious student. In FedCT instead all clients can be honest-but-curious, but their data needs to be not only protected from other clients, but also from an honest-but-curious server. Therefore, we cannot apply the Laplace mechanism that PATE uses, but instead have to protect the hard labels, which we achieve via the XOR mechanism.

(2) The privacy of distributed SGD is arguably lower than FedAvg, since sharing gradients allows not only for membership inference attacks, but even for data reconstruction attacks [2]. At the same time, FedAvg performs similar to distributed SGD in the literature [3,4]. Therefore, we have used DP-FedAvg as a baseline in our experiments. Differential privacy mechanisms are in principle compatible with most federated learning approaches, including those designed personalized federated learning [5,6]. It will be great to extensively evaluate FedCT in those scenarios, but goes beyond the scope of this work.

Weaknesses:

(1) We have discussed PATE in our related work section under semi-supervised learning, since it performs distillation via multiple teachers, instead of collaborative learning. Consequently, PATE is concerned with protecting the privacy of consensus labels, not of teachers’ predictions. Distributed distillation, instead, collaboratively trains models and naturally we compared to it - both in distributed distillation and federated co-training, clients act both as teachers and students. Since PATE shares hard predictions of teachers, we can adapt it by letting each client be a teacher. Similar to the experiments reported in [1], each client trains a model to convergence and then makes predictions on the public unlabeled dataset. A consensus is formed from all predictions that are used to pseudo-label the unlabeled dataset. A single student model is then trained on the pseudo-labeled dataset. We report the results with $m=5$ clients in Table 1.

Note that in the original experiments, a much larger number of clients is required. Therefore, we also report results for $m=100$ clients for two datasets in Table 2.

The results show that performing just a single round of training performs substantially worse than collaborative training (both distributed distillation and our proposed FedCT). Therefore, the privacy-utility trade-off of PATE is substantially worse: it achieves lower test accuracy while its privacy mechanism does not protect against a honest-but-curious server.

(2) We would like to clarify that the goal of FedCT is not to outperform federated learning in terms of test accuracy, but to improve privacy. Naturally, any gain in privacy will typically come at a loss in accuracy. Therefore, our results are actually quite astonishing. We can achieve a test accuracy comparable to FedAvg with greatly improved privacy. That is, the ability of an attacker to distinguish a private training example from an arbitrary one is as good as random guessing (VUL=$0.5$). An amazing side-effect of FedCT is that sharing hard labels not only improves privacy, but also allows us to train interpretable models, like decision trees and rule ensembles. This is not possible with FedAvg or sharing of soft labels, like in distributed distillation.

(3) We performed experiments for m=5 clients, which is typical in cross-silo scenarios that are ubiquitous in healthcare. We empirically evaluated the scalability in the number of clients on FashionMNIST in Figure 5 of our manuscript. Following your suggestion, we performed the same evaluation for the Pneumonia dataset as well and report the results in Figure 1. Similar to FashionMNIST, FedCT scales very well with the number of clients.

Figure 1: Test accuracy (ACC) of FEDCT and FEDAVG (FL) Pneumonia with |U|= 200 for various numbers of clients m.

References:

[1] Papernot, Nicolas, et al. "Semi-supervised knowledge transfer for deep learning from private training data." arXiv preprint arXiv:1610.05755 (2016).

[2] Zhu, Ligeng, Zhijian Liu, and Song Han. "Deep leakage from gradients." Advances in neural information processing systems 32 (2019).

[3] McMahan, Brendan, et al. "Communication-efficient learning of deep networks from decentralized data." Artificial intelligence and statistics. PMLR, 2017.

[4] Mills, Jed, Jia Hu, and Geyong Min. "Faster Federated Learning with Decaying Number of Local SGD Steps." IEEE Transactions on Parallel and Distributed Systems (2023).

[5] Mansour, Yishay, et al. "Three approaches for personalization with applications to federated learning." arXiv preprint arXiv:2002.10619 (2020).

[6] T Dinh, Canh, Nguyen Tran, and Josh Nguyen. "Personalized federated learning with moreau envelopes." Advances in Neural Information Processing Systems 33 (2020): 21394-21405.