Accelerated Vertical Federated Adversarial Learning through Decoupling Layer-Wise Dependencies

Thank you for your insightful comments and for recognizing the contribution of our work. Your feedback has been invaluable in enhancing the quality and clarity of our manuscript. We reply to the weaknesses and questions.

W1-Client Scale:

Experiments on More Clients. Thank you for this valuable feedback. We have conducted additional experiments on MNIST with 14 and 28 clients as shown in Table 1. These results demonstrate that DecVFAL maintains strong robustness and computational efficiency across different client configurations.

Table 1: Results with Varying Client Numbers (MNIST)

W2&Q1-Server model:

1. It is worth that server and client represent functional roles rather than physical deployment constraints. The single-layer perceptron for the server model is a standard configuration in VFL research, following established VFL frameworks where the server typically handles final classification while clients process feature-specific computations [1-3].

2. Thank you for this insightful question. We tested DecVFAL (M=6, N=8) on the MNIST dataset by deploying various server-side models: 1-layer perceptrons, 4-layer perceptrons, and 16-layer perceptrons. The experimental results reveal that as the server model becomes more complex (with more layers), it tends to slow down the entire framework's parallel efficiency. This happens because the server acts as a single module, and when it takes longer to process, all other modules must wait for its output before proceeding.

Table: Results with Different Server Model Complexities

W3-Split Position:

We acknowledge this important practical consideration. When one module contains significantly more layers, other modules must idle waiting for its output, reducing overall efficiency as demonstrated in Table 6. However, our framework provides flexibility to optimize split positions based on computational balance. We recommend treating server-client communication as a module boundary and ensuring each module's computation time is balanced and slightly greater than communication delay. This design enables modules to simultaneously send embeddings while receiving gradients, effectively eliminating communication-induced idle time and achieving maximum acceleration benefits.

Q2-Typo:

Thank you for catching this typo. We will correct "single-layer perception" to "single-layer perceptron" in line 859.

[1] LIU, Yang, et al. Vertical federated learning: Concepts, advances, and challenges. IEEE transactions on knowledge and data engineering, 2024, 36.7: 3615-3634.

[2] WEI, Kang, et al. Vertical federated learning: Challenges, methodologies and experiments. arXiv preprint arXiv:2202.04309, 2022.

[3] WANG, Ganyu, et al. A unified solution for privacy and communication efficiency in vertical federated learning. Advances in Neural Information Processing Systems, 2023, 36: 13480-13491.

We sincerely thank you for your insightful comments and constructive criticisms. Your feedback has been invaluable in improving the quality and clarity of our manuscript. Below, we address the weaknesses and respond to your questions.

W1-Privacy Analysis:

1. Delayed gradients do not introduce new privacy vulnerabilities. Delayed gradients transmit same content as immediate gradients differing only in transmission timing. Malicious participants cannot extract additional private information from delayed gradients beyond what they could already obtain from immediate gradients.

2. DecVFAL operates within the VFL-CZOFO framework [1], which provides differential privacy through zeroth-order optimization. The delayed gradients undergo the same privacy-preserving transformations as standard gradients, maintaining equivalent protection levels.

3. While AT increases communication rounds and may require a larger privacy budget compared to standard VFL-CZOFO, we will provide detailed privacy analysis in the revised version to quantify this trade-off between robustness and privacy.

W2-Model with Complex Architectures:

Decoupling easily adapts to complex architectures by treating functional blocks as module boundaries [2-6]. For complex structures like Transformers and residual networks, we can treat entire functional blocks (including their attention or residual connections) as single dynamical units. This preserves the sequential flow while enabling parallel processing. Our ResNet-18 experiments illustrate this concept: residual blocks naturally work as complete dynamical units, maintaining both the residual connections and the dynamical systems framework (footnote 4, page 7).

W3-Transmitted Content:

DecVFAL and standard VFAL transmit the same content (i.e. forward propogation for embeddings and backward propagation for gradients).DecVFAL achieves efficiency by reusing gradients to reduce the total number of communication rounds.

W4-The selection of M and N:

1. The optimal M and N values depend on many factors: dataset size and features, model structure, and training settings like learning rate and adversarial step size. Because of these complex interactions, developing an adaptive strategy is difficult.
2. Practical Selection Strategy. Empirically, M and N can be selected by referencing the choice of r in PGD-r, where M×N should be slightly larger than r, and maximizing M within the communication budget typically achieves excellent performance (only M relates to server-client communication).

[1] WANG, Ganyu, et al. A unified solution for privacy and communication efficiency in vertical federated learning. Advances in Neural Information Processing Systems, 2023, 36: 13480-13491.

[2] CHEN, Ricky TQ, et al. Neural ordinary differential equations. Advances in neural information processing systems, 2018, 31.

[3] LI, Qianxiao, et al. Maximum principle based algorithms for deep learning. Journal of Machine Learning Research, 2018, 18.165: 1-29.

[4] LI, Qianxiao; HAO, Shuji. An optimal control approach to deep learning and applications to discrete-weight neural networks. In: International Conference on Machine Learning. PMLR, 2018. p. 2985-2994.

[5] HAN, Jiequn, et al. A mean-field optimal control formulation of deep learning. Research in the Mathematical Sciences, 2019, 6.1: 1-41.

[6] WEINAN, Ee. A proposal on machine learning via dynamical systems. Communications in Mathematics and Statistics, 2017, 5.1: 1-11.

Dear Reviewer yT3W,

We appreciate your insightful review and the important questions you raised about privacy analysis, complex architectures, and parameter selection. Our rebuttal provides detailed responses to each of your concerns.

We would welcome any additional feedback or clarifications you might need, as this would help us ensure a comprehensive discussion of our work. Any additional feedback in the remaining two days would be much appreciated.

Thank you for your thoughtful evaluation.

Best regards,

The authors of Paper 10820

We sincerely thank you for your insightful comments and constructive criticisms. Your feedback has been invaluable in improving the quality and clarity of our manuscript. Below, we address the weaknesses and respond to your questions.

W1-Novelty:

Due to space limitations, please refer to W3 of Reviewer ikWZ.

W2-Baselines:

1. We selected sota AT acceleration methods (PGD, FreeAT, YOPO). They specifically target iteration complexity reduction, directly addressing the primary bottleneck in VFL scenarios: communication overhead. These baselines provide the most relevant comparison for communication-constrained environments like VFAL.

2. While we recognize recent AT developments, most focus on improving robustness through sample optimization (DOM [1]) or loss function enhancements (MART [2]) rather than addressing the computational bottleneck that VFAL faces. These approaches tackle complementary but different challenges.

3. Following your valuable suggestion, we enhanced our CIFAR-10 experiments to include recent methods like DOM [1] and MART [2] in Table 1. Importantly, our decoupling framework is compatible with these optimization strategies, allowing for combined benefits as shown in our "DecVFAL-6-2+DOM" results.

Table 1: Comparison with Recent AT Methods on CIFAR-10

W3-Deeper model:

1. Standard Benchmarks for Comparison. Our experimental settings follow well-established protocols from VFL research [6-7] and adversarial training studies (e.g., RobustBench [4]). This ensures fair comparison with existing methods and enables reproducible results.

2. Deeper Architectures Included. We have actually evaluated DecVFAL on several deeper models, including ResNeXt-50 on CIFAR-100 and Tiny-ImageNet (detailed in Section 6.1 and Appendix C.6). Since DecVFAL is designed to be architecture-agnostic, it naturally scales to modern deep networks without structural modifications.

3. Additional Experiments. We are currently conducting experiments with deeper networks on CIFAR-10 to further validate our approach. These results will be available within one week.

W4-PGD Performance:

The key insight is that DecVFAL performs significantly more adversarial updates ($M\times N$) compared to standard PGD ($r$ iterations), where typically $M\times N > r$ in our experiments (e.g., 5$\times$10=50 vs. 40 updates for MNIST, 6$\times$2=12 vs. 10 updates for CIFAR10). While individual updates use weaker gradients, the substantially higher update frequency provides more comprehensive adversarial space exploration, compensating for per-update approximation errors.

W5-Performance differences:

The primary reason for the accuracy differences is the different adversarial sample generation settings: 40 iterations on MNIST versus 10 iterations on CIFAR-10 to generate adversarial samples.

1. Slower Convergence for PGD. The high iteration count combined with MNIST's 2-layer perceptron model results in slower convergence compared to others.
2. Overfitting for FreeLB: FreeLB's gradient accumulation mechanism combined with 40 iterations causes overfitting issues, leading to poor generalization and accuracy degradation.

W6-Number of Layers in Bottom Module:

This counter-intuitive result in Table 6 is caused by error accumulation during backpropagation. While larger bottom modules do provide better gradient approximations and reduce the initial delayed gradient error, this error still persists and becomes progressively larger as it propagates backward through the network layers. This accumulation effect ultimately leads to the unstable performance we observed.

W7-Client Size and Heterogeneous Environments:

1. Experiments on More Clients. We have conducted additional experiments on MNIST with 14 and 28 clients as shown in Table 1 in W1 of Reviewer Ecn8.
2. About Heterogeneous Settings. Sample ID alignment is indeed a separate challenge in VFL that involves entity resolution and record linkage [5]. Our work focuses on the complementary problem of speeding up adversarial training when participants already have aligned data, following standard VFL assumptions.

W8-The selection of M and N:

Due to space limitations, please refer to W4 of Reviewer yT3W.

W9-Communication:

1. Decoupling does not introduce additional communication overhead, as only the server module and the top-layer client modules require true server-client communication, while other inter-module communications are purely local computations that barely affect efficiency.
2. To fully address your concern, we have included detailed communication cost statistics below to quantify the actual communication requirements and demonstrate the efficiency of our approach.

Table 2: Communication Cost

Q1-Why AT and Acceleration are needed:

1. Why AT is essential in VFL. VFL faces unique security challenges as detailed in Appendix A.3. These include third-party attackers modifying embeddings during communication and malicious participants corrupting local features. While we could use different defenses for each threat, AT offers a unified solution that builds inherent robustness against multiple attack types simultaneously.

2. The resource utilization challenge. You're absolutely right that institutions like hospitals have substantial computational resources. However, this actually shifts the bottleneck to how efficiently we use these resources. In VFAL, generating each adversarial sample requires multiple rounds of client-server communication for gradient updates. This creates significant idle time where powerful computing resources sit unused.

3. Our approach reduces communication rounds from r to M, allowing modules to process computations in parallel during communication delays. This means institutions can fully utilize their existing computational infrastructure rather than leaving it idle. Our experiments show consistent 3-10× speedup.

Q2-VFL setup:

1. The server-holding-labels setup is a standard and realistic assumption in VFL literature, as evidenced by numerous works [3,6,7], where one party naturally possesses the target labels. An example is fraud detection where a bank holds transaction labels (whether fraud) and transaction history feature while partnering with e-commerce platforms holding user shopping features.
2. It is worth that 'server' and 'client' are roles within the VFL framework rather than fixed physical entities—in practical deployments, these roles can be distributed across different participating organizations based on their data ownership and computational capabilities, with the label-holding participant typically assuming the server role for coordination purposes.
3. Our threat model considers multiple adversarial attack vectors including third-party adversaries intercepting embeddings and malicious clients, as detailed in Appendix A.3, though VFL involves other security threats like backdoor and inference attacks that fall outside our scope.
4. For scenarios with non-overlapping user sets requiring secure entity alignment, this represents another research topic in VFL [3,5], and decoupling can be integrated with existing privacy-preserving entity resolution techniques since AT operates independently of the entity alignment phase.

Typo:

Thank you for catching the typo. We will correct them and clarify DecVFAL-3-3 and DecVFAL-6-2 indicate the settings of M and N.

[1] LIN, Runqi, YU, Chaojian, HAN, Bo and LIU, Tongliang. On the Over-Memorization During Natural, Robust and Catastrophic Overfitting. In: The Twelfth International Conference on Learning Representations. 2024.

[2] WANG, Yisen, et al. Improving adversarial robustness requires revisiting misclassified examples. In: International conference on learning representations. 2020.

[3] LIU, Yang, et al. Vertical federated learning: Concepts, advances, and challenges. IEEE transactions on knowledge and data engineering, 2024, 36.7: 3615-3634.

[4] CROCE, Francesco, et al. RobustBench: a standardized adversarial robustness benchmark. In: Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).

[5] HUANG, Lingxiao, et al. Coresets for Vertical Federated Learning: Regularized Linear Regression and $K$-Means Clustering. Advances in Neural Information Processing Systems, 2022, 35: 29566-29581.

[6] WEI, Kang, et al. Vertical federated learning: Challenges, methodologies and experiments. arXiv preprint arXiv:2202.04309, 2022.

[7] WANG, Ganyu, et al. A unified solution for privacy and communication efficiency in vertical federated learning. Advances in Neural Information Processing Systems, 2023, 36: 13480-13491.

Dear Reviewer 2ipB,

For your concern "W3-Deeper model", We are very excited to share new experimental results that directly address the concern about model depth. We have completed experiments on CIFAR-10 using WideResNet-70-16, a significantly deeper architecture with 70 layers. DecVFAL achieves comparable robust accuracy to PGD while providing 2.46× speedup. The results demonstrate that DecVFAL maintains its effectiveness on deep networks:

We hope our rebuttal can address your concerns. If you have any questions, we would be very happy to discuss them with you in depth.

Dear Reviewer 2ipB,

Thank you for your detailed review and the specific concerns you raised. We have provided extensive responses addressing novelty, baselines comparison, deeper network evaluation, and communication analysis. We've also included additional experimental results on WideResNet-70-16 as requested.

With two days left in the discussion phase, we'd welcome any further thoughts on our responses. This would allow us to address your concerns more thoroughly.

Thank you for your careful evaluation.

Best regards,

The authors of Paper 10820

Dear Reviewer 2ipB,

Thank you for your positive response and for increasing your score. Your constructive feedback has been invaluable in strengthening our work, and we appreciate your thorough engagement throughout this review process.

Should you have any further questions or concerns, we would be more than happy to address them. We wish you continued success in your research endeavors.

Best regards,

The authors of Paper 10820

Dear Reviewer 2ipB,

Thank you for your thoughtful engagement with our work. We respectfully address your concern about "marginal novelty" by clarifying our significant technical and theoretical contributions that extend well beyond incremental improvements.

Fundamentally Different Application of Decoupling

While previous decoupling works focused on parameter updates during standard training, DecVFAL is the first approach to apply decoupling to adversarial sample generation—the most computationally intensive component of AT. This represents a fundamental innovation rather than a marginal adaptation.

Our key insight is that adversarial sample generation offers unique advantages for decoupling that are unavailable in standard training. Since model parameters remain frozen during adversarial sample generation, our approach enables true parallel processing rather than the approximate parallelism of forward propagations in previous work. This fundamental difference allows us to achieve substantially greater efficiency gains while maintaining robustness guarantees.

To demonstrate the standalone value of our approach beyond VFL, we evaluated DecVFAL in a centralized (non-VFL) setting with a single client, simulating local AT on CIFAR-10. Even when applied to standard PGD-based AT, our method achieves significant improvements:

Moreover, VFL's natural vertical partitioning perfectly aligns with our layer-wise decoupling mechanism, enabling even greater efficiency gains in distributed settings, as confirmed by our experimental results in the paper.

Unique Technical Challenges

Applying decoupling to adversarial sample generation in VFL introduces unprecedented challenges not addressed in prior work:

1. Preserving Complex Optimization Structure: Unlike standard training, AT involves intricate minimax optimization that alternates between adversarial sample generation and model parameter updates. Our framework carefully maintains this delicate balance while introducing parallelization. Thsi is a non-trivial work that required novel algorithmic design.

2. Managing Multi-source Approximation Errors: The distributed nature of VFL compounds approximation challenges significantly. The VFL framework inherently requires communication-efficient techniques, necessitating the introduction of compression and gradient approximation methods that generate multiple sources of approximation errors. These accumulated errors pose a substantial risk of degrading the robustness of AT. Our framework innovatively balances gradient quality and computational efficiency through carefully designed approximation strategies that preserve model robustness while achieving substantial speedups.

We sincerely appreciate your insightful comments and constructive feedback, which have been invaluable in improving the quality and clarity of our manuscript. Below, we provide detailed responses to the identified weaknesses and questions.

W1&Q1-Real-World Dataset:

Thank you for the recognition of this limitation. To address your concern, we added 2 real-world datasets: COVID-19 Image Data Collection [1] and Credit Card Fraud Detection [5] to further validate our work. VFL Setup: we set 2 clients which each uses ResNet-18 for COVID-19 and MLP for Credit, one server employs 1-layer classifier. DecVFAL achieves 3.72× speedup and 2.35× speedup respectively compared to standard PGD, while maintaining superior robustness.

Table 1: Results of Robust Training on COVID-19 dataset [1]

Table 2: Results of Robust Training on Credit dataset [5]

W2&Q2-Impact of the number of modules:

1. The trade-off between module number and accuracy is inherent to module decoupling and must be carefully balanced. Our theoretical analysis (Remark 1) and empirical results (Table 5) consistently demonstrate that increasing module numbers $\mathcal{M}_K$ degrades performance. This outcome is consistent with prior decoupled training work [2-4].

2. The practical strategies to mitigate this include (1) against excessive module partitioning, (2) align the execution time of all partitioned modules to maximize parallelization benefits.

W3-Novelty:

Thank you for recognizing the non-trivial contributions of our framework. We want to emphasize the unique challenges our work addresses.

1. Technical Innovation: While we build upon fundamental techniques, applying decoupling to communication-efficient VFL presents distinct challenges. Our approach uniquely combines decoupling with AT and VFL-specific methods like Zero-Order Optimization and compression [8]. The key innovation lies in decoupling adversarial sample generation with frozen parameters - a departure from prior work [2-4] that only decouples parameter updates. This enables true concurrent forward/backward propagation and addresses the computational bottleneck in adversarial sample generation, achieving substantial 3-10× acceleration for VFAL.

2. Theoretical Contribution: The simultaneous presence of multiple gradient approximations creates analytical complexity not addressed in existing literature. Our framework must account for three interacting approximation sources: delayed gradients (from decoupling), compressed gradients (from embedding compression), and estimated gradients (from ZOO). We provide the first convergence analysis that captures how these approximation errors propagate and interact within VFAL systems, requiring careful treatment of their cumulative effects.

Q3-Asynchronous Settings Impact:

We think that full asynchronous VFL updating presents significant convergence challenges. AT operates through a minimax optimization that alternates between adversarial sample generation and model parameter updates. This requires freezing model parameters during adversarial sample generation, followed by synchronized parameter updates using the generated samples. Asynchronous settings would disrupt this minimax execution order, potentially leading to scenarios where model parameters are updated before optimal adversarial samples are identified.

[1] MAGUOLO, Gianluca; NANNI, Loris. A critic evaluation of methods for COVID-19 automatic detection from X-ray images. Information fusion, 2021, 76: 1-7.

[2] JADERBERG, Max, et al. Decoupled neural interfaces using synthetic gradients. In: International conference on machine learning. PMLR, 2017. p. 1627-1635.

[3] HUO, Zhouyuan, et al. Decoupled parallel backpropagation with convergence guarantee. In: International Conference on Machine Learning. PMLR, 2018. p. 2098-2106.

[4] HUO, Zhouyuan; GU, Bin; HUANG, Heng. Training neural networks using features replay. Advances in Neural Information Processing Systems, 2018, 31.

[5] CARCILLO, Fabrizio, et al. Combining unsupervised and supervised learning in credit card fraud detection. Information sciences, 2021, 557: 317-331.

[6] CROCE, Francesco, et al. RobustBench: a standardized adversarial robustness benchmark. In: Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).

[7] LIU, Yang, et al. Vertical federated learning: Concepts, advances, and challenges. IEEE transactions on knowledge and data engineering, 2024, 36.7: 3615-3634.

[8] WANG, Ganyu, et al. A unified solution for privacy and communication efficiency in vertical federated learning. Advances in Neural Information Processing Systems, 2023, 36: 13480-13491.

Dear Reviewer ikWZ,

Thank you for your thorough review and constructive feedback on our paper. We have addressed your concerns regarding real-world datasets, bias term trade-offs, and asynchronous settings in our detailed rebuttal.

With the discussion phase ending in two days, we'd greatly appreciate any follow-up comments you might have.

Thank you for your time and valuable insights.

Best regards,

The authors of Paper 10820