VFLAIR: A Research Library and Benchmark for Vertical Federated Learning

Response to Weakness 1 and Question 1

Thank you for your constructive suggestions. We have included 3 new communication protocols, including Quantizate [1], Top-k [1] and CELU-VFL [2]. We also added FedGNN to benchmark its performance on Cora dataset. Details on these experimental settings including model partition, hyperparameters are shown in Tab. 10 to 12 in Appendix H. Results are all included in Tab. 6 for communication protocols and in Tab. 7 for Cora dataset with FedGNN, both in Section 6.1. For quicker access, please refer to our reply in "Response to Weakness 4 and Question 3 for Reviewer z5tx" (communication protocols) and “Response to Weakness 3 and Question 3 for Reviewer z5tx” (Cora dataset with FedGNN, Tab(1) in the response). We do not show them here again to avoid repeatitiveness and redundancy.

Response to Weakness 2 and Question 2

To demonstrate that our platform is lightweight, we compare it with FATE, one of most widely used FL platform supporting a broad range of VFL functionalities, on their system requirements for deployment. According to FATE (link), deploying a stand-alone version of the FATE framework requires at least a 8 core CPU with 16G memory and 500G hard disk and the downloaded docker package for deployment is of size 4.92G for version 1.7.1.1. However, for our VFLAIR, a 1 core CPU with less than 4G memory and less than 4.0G hard disk is required for installation.

(4) Comparision between FATE and VFLAIR

Response to Weakness 3 and Question 3

Thank you for your suggestions. We have added Criteo and Avazu into our experiments, and results are included in Tab. 7 in Section 6.1. For quicker access, please refer to our reply in “Response to Weakness 3 and Question 3 for Reviewer z5tx” (Tab(1) in the response). We do not show them again here to avoid repeatitiveness and redundancy.

Response to Weakness 4 and Question 4

Thanks for the suggestion. We prioritized benchmarking non-cryptographic defense methods because they are emerging more rapidly in recent years due to their capability of defending various types of atttacks, whereas cryptographic techniques typically protect data in-transit during training, can not defend backdoor attacks or inference-time attacks effecitively while incurring significant computation and communiction cost to run. The benchmark of cryptographic techniques focus more on efficiency. Nevertheless, we think that the combination of cryptographic and noncryptographic methods would be an interesting research direction and potential solution for practical deployment.

For VFLAIR, we have already included one of most commonly used cryptographic techniques, i.e. Paillier Encryption, in tree-based VFL and evaluated its impact on model performance and computation efficiency in Table. 5 in Section 6. We are currently implementing Encryption techniques to NN-based VFL and also plan to add more advanced privacy-preserving cryptographic methods to our VFLAIR in the near future.

Response to Weakness 5 and Question 5

Thank you for your suggestions. We have now formally added communication rounds (#Rounds) and amount of exchanged inforation each round (Amount) for the evaluation of communication efficiency, as well as Execution Time (Exec. Time) for evaluating computation efficiency in Section 5.3 Evaluation Metrics. Further we applied these metrics in our results in Section 6.1 when comparing multiple communication protocols in Tab. 6, and when comparing the computation efficiency in Tab. 5.

Reference

[1] Castiglia, Timothy J., et al. "Compressed-vfl: Communication-efficient learning with vertically partitioned data." International Conference on Machine Learning. PMLR, 2022.

[2] Fu, Fangcheng, et al. "Towards communication-efficient vertical federated learning training via cache-enabled local updates." arXiv preprint arXiv:2207.14628 (2022).

Dear reviewer, do our responses to your questions as well as the updated paper address your concerns? We have added federated graph neural network as advanced model (Tab. 7 in Section 6); quantization, Top-k compression as well as CELU-VFL for additional communication protocols (Tab. 6 in Section 6); recommandation datasets including Criteo and Avazu (Tab. 7 in Section 6); as well as more communication and computation effectiveness evaluation metrics to our VFLAIR platform and benchmark experiments (in Section 6) according to your reviews. We have also added comparition to FATE to show that our platform is lightweight (in Appendix D). We are currently working on supporting Paillier Encryption in Linear Regression models in our platform. We hope the above have solved your concerns and we are still looking forward to your further replies. We will remain open and available if there is any further concerns.

Response to Weakness 1

Thank you for the suggestion. As a research library designed to facilitate research development in the VFL, our VFLAIR incorporated all the essential components of a VFL system, including 2 model partition strategies, 5 communication protocols (3 added during discussion), 1 encryption protocol, 13 datasets (4 added during discussion) of different modality and data partition methods, 29 model architectures, in addition to 11 attacks and 8 defenses. In addition, our VFLAIR can be easily used and extended to perform evaluations on various aspects of a VFL system, including model performance, computation and communication efficiency under a broad range of settings.

Therefore, we consider our VFLAIR a comprehensive VFL library and will continue to add more state-of-art privacy-preserving and communication-efficient methods in the future. As for our benchmark evaluations, we have further refined and completed new evaluations on model performance of new datasets (Tab. 7 in Section 6.1), communication and computation efficiency of various communication strategies (Tab. 6 in Section 6.1). Therefore, although comprehensive evaluation of attack and defense is an outstanding feature of our benchmark, we think other aspects should not be overlooked. So we would prefer to keep our title unchanged.

Response to Weakness 2

We validate the effectiveness of our proposed DCS metrics from the following 3 aspects.

• Robustness. The results of the C-DCS rankings are generally consistent across all datasets (MNIST, CIAFR10 and NUSWIDE) in Tabs. 14 to 16 under the same setting. Also, the C-DCS ranking of the defense methods are still generally consistent under different model partition setting and communication protocols in Tabs. 14, 17 and 18. Below we plot the mean and standard derivation (std) of the ranking of each defense in the above 5 ranking tables with different defense hyper-parameter shown in this figure (link). As shown in this figure, the ranking is relatively stable and the std is small across multiple settings for all the defenses. Finally, as shown in Figs. 4, 14 and 15, the C-DCS ranking is also generally stable with the change of $\beta$, especially when $\beta \leq 0.7$. All these demonstrate the good stability and credibility of our proposed C-DCS metrics.
• Sensitivity. We demonstrate that the changes in DCS value is a good indicator of relative defense capability with the following evidence: (1) From Tabs. 14 to 16, we show that T-DCS$_{FR}$ (T-DCS for feature reconstruction attacks) values are much lower than the T-DCS values of other attack types, indicating inferior defense capabilities under feature reconstruction attacks. This phenomenon is consistent with the fact that feature reconstruction attacks, especially TBM which uses auxiliary data for reconstruction, result in much stronger attack capabilities (shown in Figs. 3, 11 and 12). Here we also show the visualization of the reconstructed images of TBM under different defense methods in this figure (link). It is apparent that although the reconstructed images are noisy, the contour can still be seen easily. This further validates the effectiveness of our proposed T-DCS in measuring the relative defense strength of different types of attacks. (2) The differences in DCS scores between splitVFL and aggVFL settings (in Section 6) are also consistent with the human domain knowledge that splitVFL is less vunerable to attacks compared to aggVFL, which is consistent with findings in [1], demonstrating the effectiveness of the metrics.
• Alignment with human intuition. By definition (Eq. (1)), our DCS score measures its closeness to a perfect defense (Fig. 2). Intuitively, a higher DCS score represents a defense that achieves a lower AP at a higher level of MP compared to others. This definition aligns with the defense target, i.e. limiting the AP to a lower level while maintaining a higher level of MP.

Reference

[1] Yan Kang, et al. "A framework for evaluating privacy-utility trade-off in vertical federated learning". arXiv preprint arXiv:2209.03885, 2022

Dear reviewer, do our responses to your questions as well as the updated paper address your concerns? We have added a workflow figure in (in the appendix and the link in the response), testified the effectiveness of our DCS evaluation (in the response and in the appendix) and reorganized the result section (Section 6) and related work section (Section 2) according to your reviews. We hope the above have solved your concerns and we are still looking forward to your further replies. We will remain open and available if there is any further concerns.

Dear reviewer, we are still anxiously looking forward to your replies to our responses. Time is limited with less than 5 hours till the end of the discussion period. We are curious to know if our responses to your questions as well as the updated paper have already addressed your concerns. We have put a lot of effort into working to address your concerns and to improve our work. We summarize our modifications with respect to your reviews as below:

• We testified the effectiveness of our DCS evaluation in our response from 3 different aspects to answer your question, including Robustness, Sensitivity and Alignment with human intuition. We have also included these analyses in our updated paper in Section 6 and Appendix I.2.6.
• We have added a workflow figure in (in appendix D and a link for quick access in the response).
• We reorganized the result section (Section 6) in a more systematic way according to your reviews. We also modified the related work section (Section 2) according to your helpful suggestion.
• We carefully considered your comment on the paper title and clearly presented the reasons why we choose to kept it unchanged.

We hope the above have solved your concerns and we are still looking forward to your further replies. We will remain open and available if there is any further concerns.

Response to Weakness 1 and Question 1

In order to better illustrate the systematic design of VFLAIR, we add Fig. 10 in Appendix D (see also this figure (link)) to demonstrate the key modules and workflow of VFLAIR. We also add a step by step guidance for using and extending VFLAIR in Appendix C. In summary, building on a modularized design of components covering all essential aspects of a VFL system, using VFLAIR only requires 5 easy steps as shown in Fig. 8 in Appendix C (see also this figure (link)), with a single line to run: "python main_pipeline.py --configs my_config".

For extending VFLAIR, we summarized the necessary steps and also included 2 demo cases ("how to add a new attack" and "how to add a new local model") in Appendix C. In summary, extending VFLAIR requires only modifying the configuration file, implementing code by inheriting corresponding classes (e.g. attacker) or by adding the model architecture and run.

Please see Appendix C and D for more details.

Response to Weakness 2 and Question 2

Thanks for the suggestions. We have added and highlighted the summary of insights at the beginning of each paragraph throughout section 6. Please see our modified manuscript.

Response to Weakness 3 and Question 3

Thanks a lot for your suggestions. We first need to clarify that, different from horizontal federated learning (HFL) where datasets are partitioned by samples, "non-IID data" distribution is not considered in VFL because datasets are partitioned by features. That said, we did consider "unequal split of features among each party" in our experiments with justifiable partitioning methods. For example, as we already included in our submission, NUSWIDE is a cross-modal dataset which contains an image field of 634 features and a text field of 1000 features, which are naturally partitioned into two parties by modality, resulting in an unbalanced feature set among parties. This also aligns with real-world data distribution where text and its paired image are held by different parties.

Despite the rapid growth of real-world VFL applications, it is still difficult to collect truly distributed real-world dataset due to data privacy and regulations. However, we have additionally added Criteo, Avazu, Cora and News20 into our benchmark results in Tab. 7 in Section 6.1 (and below). They are considered real-world datasets for typical VFL applications (Criteo and Avazu for click through rate prediction in advertising, Cora for node classification in citation network, and News20 for classification of news data). As shown in the table below, we adopt partition strategies aligning with human observation of features, resulting in unbalanced features among parties (see also Tab.10 in Appendix H for complete dataset and partition details).

(1) Tab. 7 (in paper). MP, communication rounds (#Rounds) for reaching specified MP with 4 real-world datasets of NN-based VFL with FedSGD communication protocol.

(2) Feature (dataset) partition strategies that are unequal. (p): passive party, (a): active party for column 'Number of Features'.

Dear reviewer, do our responses to your questions as well as the updated paper address your concerns? We have added more unequal dataset partition methods (in Tab. 7 in Section 6); more communication protocols (in Tab. 6 in Section 6) like quantization, Top-k compression and CELU-VFL to our VFLAIR platform and benchmark experiments according to your reviews. We have also added a step by step user guidance as well as 2 demos on how to extend VFLAIR in the appendix (Appendix C). Moreover, we have reorganized Section 6 and highlighted the insights of each paragraph in our updated paper. We hope the above have solved your concerns and we are still looking forward to your further replies. We will remain open and available if there is any further concerns.

Thanks for your response. The revision has addressed most of my concerns. The paper is more clear now. I have raised my score to 8. For the benchmark, it'd be great if the authors could make it a Python library so that researchers can easily import the approaches for more flexible usage.

Dear reviewers, AC and PC:

We have updated our paper and supplementary material for adding highlights on the modifications we made to our paper. We look forward to your further replies. Thanks.