PubSub-VFL: Towards Efficient Two-Party Split Learning in Heterogeneous Environments via Publisher/Subscriber Architecture

W1: We agree that PS and asynchronous training techniques have been previously explored in FL. However, the novelty of our work lies not in reusing these mechanisms in isolation, but in the system-level integration and adaptation of them to a two-party VFL setting, which presents unique challenges not addressed by prior work. Specifically, our novelty lies in the following two aspects:

1. Tightly-Coupled ID Alignment in VFL: Unlike HFL, two-party VFL requires strict alignment of instance IDs across feature spaces. Prior AVFL or PS-based designs fail to decouple this step from training, resulting in latency and straggler effects. Our use of Pub/Sub effectively breaks this bottleneck, i.e., an architectural insight not previously explored in this setting.

2. Hierarchical Asynchrony: While asynchronous updates are known, our hierarchical design balances flexibility and convergence, tailored specifically for the computation-communication imbalance inherent in two-party VFL with resource heterogeneity.

W3: We acknowledge that our theoretical analysis adopts standard assumptions such as strong convexity and smoothness. These are indeed idealized and may not capture the full complexity of deep learning models used in practice. However, our goal with this theoretical component is to establish a foundational convergence guarantee under the proposed hierarchical asynchronous mechanism with privacy noise injection. Our analysis provides initial insights into the stability and convergence behavior of the systems.

Moreover, this theoretical groundwork is aligned with common practice in FL literature (e.g., [36], [38]), where convex assumptions are used to gain intuition about the impact of asynchrony and staleness. We believe this offers useful guidance for system design, even if the practical models are non-convex. We agree that extending the analysis to non-convex settings or realistic deep networks would be valuable, and we consider this an important direction for future work.

Q2: The system profiling and optimization in PubSub-VFL are designed with practical robustness in mind, and while they are primarily calibrated during the offline planning phase, the framework incorporates inherent mechanisms to handle runtime changes in resource availability or network bandwidth without compromising core functionality.

First, the system profiling focuses on capturing representative baseline characteristics rather than rigid static values. This baseline provides a stable foundation for initial hyperparameter optimization, but the framework does not rely strictly on these values in runtime.

Second, PubSub-VFL integrates dynamic runtime safeguards (i.e., the buffer mechanism, the waiting deadline mechanism, and the hierarchical asynchronous design) that implicitly adapt to fluctuations, e.g., indefinite blocking, sudden CPU overload, and less sensitivity to transient resource imbalances. Moreover, the offline optimization model (Section 4.3) already accounts for expected variability in resources and bandwidth by bounding feasible hyperparameters. This ensures that even if runtime conditions deviate slightly from the baseline, the pre-optimized configurations remain within a robust operating range.

Q3: Indeed, the introduction of differential privacy noise can lead to a degradation of system performance. However, our proposed embedding-level GDP (see Appendix C) can achieve a better privacy-utility trade-off (see page 9, line 346, and Figure 5 for details). To address the utility degradation at low privacy budgets when using DP in PubSub-VFL, we provide detailed clarification based on our experimental results (especially Fig. 5) and additional evaluations using EIAs:

• Accuracy: Even at low privacy budgets (e.g., $\mu = 0.1$), the model’s accuracy remains stable, with only a marginal drop compared to scenarios without DP ($\mu = +\infty$). For example, on the Bank dataset, the AUC decreases by less than 2% when $\mu$ is reduced from $+\infty$ to 0.1, demonstrating that PubSub-VFL maintains strong predictive performance even under strict privacy constraints.

• Computational Efficiency: CPU utilization shows minimal degradation across all privacy budgets, staying above 85% for most settings.

• Communication Cost: As expected, lower privacy budgets (smaller $\mu$) lead to increased communication cost. This is because stricter privacy requires injecting more Gaussian noise into embeddings, increasing the number of training rounds. However, this increase is manageable and does not negate the overall efficiency gains of PubSub-VFL compared to baselines.

• EIA: At high privacy budgets ($\mu \geq 4$), the ASR remains relatively high (e.g., ~30% on the Synthetic dataset), indicating that embeddings are vulnerable to inversion when noise is minimal. At low privacy budgets ($\mu \leq 1$), the ASR drops to near 0%, demonstrating that the injected DP noise effectively mitigates EIA.

Thank you for authors' efforts. After reading authors' rebuttal, W2 has been resolved. But I still have my concerns regarding W1 and W3. I have raised my score to 3 (Borderline reject).

W3&Q3: To address the concerns about Equation (6) and its simplification under equal worker assignment, we clarify the notation, assumptions, and derivation as follows:

1. Original Form of Equation (6)

Equation (6) in the paper defines the forward pass computation delay for the active party ($P_a$) and passive party ($P_p$) as:

• Notation:
  • $T_f^{(a)}(B)$ / $T_f^{(p)}(B)$: Time for forward pass on a batch of size $B$ for $P_a$ / $P_p$.
  • $\lambda_a$, $\lambda_p$: Proportionality constants capturing base computational overhead (e.g., per-sample processing time).
  • $B^{\gamma_a}$, $B^{\gamma_p}$: Batch size term, where $\gamma_a$, $\gamma_p$ model how computation scales with batch size (e.g., $\gamma \approx 1$ for linear scaling).
  • $w_a$ / $w_p$: Number of workers for $P_a$ / $P_p$.
  • $c_{a,j}$ / $c_{p,i}$: CPU cores allocated to the $j$-th worker of $P_a$ / $i$-th worker of $P_p$.
  • $\sum_{j=1}^{w_a} c_{a,j} \leq C_a$ / $\sum_{i=1}^{w_p} c_{p,i} \leq C_p$: Total cores allocated to workers cannot exceed the total CPU cores available to $P_a$ ($C_a$) or $P_p$ ($C_p$).

2. Simplification Under Equal Worker Assignment

For simplicity, if cores are equally allocated to workers, Equation (6) simplifies to:

This simplification is mathematically justified:

• Under equal allocation, each worker of $P_a$ gets $c_{a,j} = \frac{C_a}{w_a}$ cores (since total cores $\sum_{j=1}^{w_a} c_{a,j} = C_a$).
• Substituting $\sum_{j=1}^{w_a} c_{a,j} = C_a$ into the original Equation (6) and utilizing Data Parallel Processing, we have:

3. Key Assumptions in the Derivation

The simplification relies on two explicit assumptions:

1. Equal Core Allocation: Workers within a party share CPU cores uniformly ($c_{a,j} = C_a / w_a$ for all $j$). This avoids complex per-worker core tuning and simplifies the model.
2. Fixed Total Cores: The total cores allocated to workers do not exceed the party’s maximum available cores ($\sum c_{a,j} = C_a$). This ensures resource constraints are respected.

These assumptions are reasonable for offline system profiling (Section 4.2), where the goal is to approximate computation delays for hyperparameter optimization (e.g., choosing $w_a$ and $B$ via dynamic programming).

W4&Q4: Sharing system profile information in PubSub-VFL poses minimal additional privacy risks due to the nature of the data involved and the framework’s design. System profiles primarily include infrastructure metadata, such as CPU core counts, memory limits, and computational delay parameters, rather than sensitive data like user records, personal identifiers, or business secrets. These technical details, which focus on resource allocation and model structure, do not reveal information about the content of the data being processed (e.g., financial transactions or medical records).

Moreover, the optimization process relies on local profile computations: each party calculates its own feasible hyperparameters (worker counts, batch sizes) based on internal resources without exposing raw profile data to the other party, ensuring sensitive infrastructure details remain private. Since this metadata lacks the granularity to reveal sensitive patterns or identifiers, and existing privacy protocols (e.g., differential privacy for embeddings) already protect actual data, sharing system profiles does not compromise core privacy guarantees or violate regulations like GDPR, which target personal data rather than infrastructure specifications.

W1&Q1: This paper does not propose dynamically adjusting the number of workers during training. Instead, it determines the optimal number of workers $w_a$ for the active party, $w_p$ for the passive party) in the offline system planning phase (Section 4.3) using a dynamic programming algorithm. This optimization is based on precomputed system profiles (e.g., CPU cores, memory constraints, and delay models) and is fixed before training starts. The paper explicitly states that hyperparameters like worker counts are selected to balance computational loads ahead of execution (Algorithm 2), with no mechanism for real-time adjustments during training.

Even with fixed computational resources in a two-party scenario, pre-negotiating worker counts based on detailed system profiles (rather than static, pre-agreed values) remains critical for several reasons:

1. Hidden Resource Heterogeneity in "Fixed" Environments
   Fixed resources (e.g., a 64-core CPU split between parties) do not imply uniform performance. Factors like background processes, memory bandwidth, or cache utilization can cause real-time variations in effective computational power. The paper’s system profiling (Section 4.2) captures these nuances by modeling forward/backward pass delays as functions of worker counts and batch sizes (Eqs. 7–9). For example, Table 2 shows that 8 workers (vs. 4, 10, or 50) yield the highest CPU utilization (88.04%) and lowest runtime for the synthetic dataset—this optimal count cannot be pre-negotiated without empirical profiling, even with fixed total cores.

2. Load Imbalance Mitigation
   In two-party VFL, the active party incurs additional computational overhead (e.g., top model training, loss calculation) compared to the passive party (Section 3). Pre-negotiating a static worker ratio (e.g., 1:1) ignores this inherent asymmetry. The paper’s offline optimization dynamically balances $w_a$ and $w_p$ to minimize the maximum iteration time between parties (Eq. 11), ensuring neither party is idle. For instance, in resource-heterogeneous scenarios (CPU core ratios 50:14), PubSub-VFL’s optimized worker counts maintain 87.42% CPU utilization, while static ratios (as in baselines) drop to 42.12% (Section 5.2).

3. Alignment with Real-World Deployment Constraints
   Even with fixed resources, practical deployments involve trade-offs between batch size, memory limits, and worker efficiency. The paper’s DP algorithm (Algorithm 2) integrates memory constraints (Eq. 12) to avoid out-of-memory errors, a detail static pre-negotiation might overlook. For example, Table 3 shows that a batch size of 256 (paired with 8 workers) achieves 91.07% CPU utilization, while larger batches (512, 1024) suffer from memory bottlenecks—this balance requires profiling, not just pre-negotiation.

W2&Q2: First, Equation (5) defines the dynamic synchronization interval $\Delta T_t$ for the intra-party semi-asynchronous mechanism:

This formula adjusts the synchronization frequency based on training progress (epoch $t$), not on an assumption that performance unconditionally improves with more rounds. The design logic is pragmatic:

• Early in training (when the model is far from target accuracy), $\Delta T_t$ is small, meaning more frequent synchronization. This ensures stable gradient updates and prevents divergence from stale parameters.
• As training progresses (when accuracy approaches the target), $\Delta T_t$ increases, reducing synchronization overhead to accelerate fine-tuning.

Second, PubSub-VFL includes mechanisms that mitigate overfitting and ensure consistent performance:

• Convergence Guarantees: Appendix D formally proves that PubSub-VFL converges stably under standard assumptions (smoothness, strong convexity, bounded gradient variance). The convergence bound (Theorem D.1) shows that the expected optimality gap decreases over time, with an error floor determined by noise (stochastic gradients and DP noise), not by unchecked overfitting.

• Empirical Generalization: Experiments across five datasets (Section 5.2) demonstrate that PubSub-VFL maintains or improves accuracy compared to baselines (e.g., 96.54% AUC on the Bank dataset vs. 94.54% for VFL). Even on large-scale synthetic data (1M samples), it achieves 92.87% AUC, with no signs of overfitting (e.g., no divergence between training and test performance, which would indicate overfitting).

• Implicit Regularization: The integration of Differential Privacy (DP) adds controlled noise to embeddings (Section 4.1), which acts as a regularization mechanism. DP noise reduces the model’s ability to memorize training data specifics, a known strategy to prevent overfitting in sensitive settings.

W1&W2: We will make corresponding changes in future revisions.

W3: Thank you for your suggestion; it is an interesting proposal, and we will carefully consider it in the revised version.

W4: Yes, two-party VFL remains a highly relevant and practically deployed paradigm in collaborative machine learning, particularly in privacy-sensitive domains where different entities possess complementary feature spaces over the same set of users. It is widely supported by industrial-grade platforms such as FATE, PaddleFL, and IBM FL, which have production deployments in sectors like finance and healthcare.

Q2: In addition to the four real-world public datasets (Energy, Blog, Bank, and Credit), we have already included a large-scale synthetic dataset in our evaluation (see Table 1 and Figures 3–5), which contains 1 million samples and 500 features, to simulate real-world big data conditions. This dataset was designed to stress-test both computational efficiency and communication scalability of PubSub-VFL under heavy workloads.

To further evaluate scalability, we introduce the Criteo 1TB Click Logs dataset [5], a widely used industrial benchmark for online advertising and recommendation systems. It contains ~4.5 billion samples with 39 features (13 numerical, 26 categorical, and high-dimensional after one-hot encoding), representing real-world big data characteristics (massive samples and sparse features). Following the evaluation metrics (AUC for classification, runtime, CPU utilization, waiting time per epoch, and communication cost) in our paper, the table below compares PubSub-VFL with baselines:

[5] https://ailab.criteo.com/download-criteo-1tb-click-logs-dataset/

Key Observations:

• Accuracy: PubSub-VFL achieves the highest AUC (82.15%), outperforming baselines by 0.7–1.2%, indicating robustness in large-scale, sparse data scenarios.
• Efficiency: It reduces runtime by ~3× compared to AVFL-PS (21.5h vs. 6.8h) and ~7× compared to VFL (48.6h vs. 6.8h), thanks to hierarchical asynchrony and Pub/Sub decoupling.
• Resource Utilization: CPU utilization reaches 90.8%, significantly higher than baselines, demonstrating effective load balancing in heterogeneous large-scale environments.
• Communication Cost: 450GB is ~40% lower than AVFL-PS, due to optimized channel management and reduced stale updates.

These results validate PubSub-VFL’s scalability and efficiency in real-world big data settings.

Q3: The intra-party semi-asynchronous mechanism was primarily designed to address resource heterogeneity within each party (e.g., workers with varying compute capacity) and to reduce idle time caused by synchronous intra-party updates in PS setups. In traditional synchronous PS-based VFL, faster workers must wait for the slowest one before proceeding, leading to poor CPU utilization and degraded efficiency, especially in heterogeneous environments. Our motivations are summarized as follows:

• Mitigate straggler effects within each party’s workers.
• Increase pipeline parallelism and throughput without compromising convergence.
• Complement the inter-party asynchronous mechanism enabled by the Pub/Sub architecture.

Technical Trade-offs and Design Decisions:

While full asynchrony can maximize efficiency, it may introduce gradient staleness, hurting convergence stability. To strike a balance, we adopt a semi-asynchronous update scheme, where the synchronization interval $\Delta T_t$ is dynamically adjusted based on training progress:

This adaptive control mechanism ensures that:

• Early in training, $\Delta T_t$ is small → frequent synchronization, improving stability.
• Later in training, $\Delta T_t$ increases → more relaxed synchronization, enhancing efficiency.

Empirical Insights: As shown in Tables 2 and 3, and Figures 3–4, this mechanism enables:

• Higher CPU utilization (up to 91.07%)
• Lower waiting times, even with heterogeneous resources
• Comparable or improved accuracy, indicating convergence is not compromised

Thank you for your response, which addressed most of my concerns, especially regarding the transition from two-party to multi-party settings. This makes me really appreciate the practical nature of this work, so I plan to raise my score to 5.

Aside from that, I have a small question: when you mentioned DP, you meant a dynamic programming algorithm, right? Not differential privacy? Since you already ensure privacy with PSI, there’s no need for differential privacy, correct?

W1&Q1: While resource and data heterogeneity are often associated with multi-party federated learning, this paper demonstrates that such heterogeneity remains highly relevant even in two-party VFL scenarios. We justify this by showing that computational resource disparities (e.g., CPU core ratios like 50:14) and imbalanced feature dimensions (e.g., 50:450) between the two parties can significantly impact training latency and CPU utilization. To address this, the proposed PubSub-VFL explicitly models and optimizes for these imbalances through a privacy-preserving hyperparameter tuning strategy that adapts to each party's system profile. Therefore, the focus on two-party VFL is not a limitation but a targeted demonstration that resource and data heterogeneity can still severely affect efficiency, even in minimal collaborative settings, and merit dedicated architectural and algorithmic interventions.

W2-3&Q2: In this paper, we aim to demonstrate the advantages of PubSub-VFL by comparing the performance, resource utilization, and communication overhead of different abstract VFL frameworks. Although abstract AVFL frameworks may involve different implementations (such as [36-39]), their core concept remains the design of asynchronous mechanisms. To further illustrate this point, we follow the experimental setup of the original manuscript and conduct a set of comparative experiments on the Blog dataset, as shown in the table below. The experimental results show that PubSub-VFL still has advantages over the frameworks in [36-39].

W4&Q3: While the primary goal of PubSub-VFL is to improve computational efficiency through hierarchical asynchronous design and system-profile-based optimization, the observed accuracy improvements in Tables 1 and 6 are not incidental. These gains can be attributed to several factors:

1. Reduced Waiting Time and Staleness: By eliminating inter-party and intra-party synchronization bottlenecks, the system reduces idle time and stale updates, allowing for more timely and consistent parameter updates. This helps improve convergence stability, especially in heterogeneous environments.

2. Better Hyperparameter Configuration: The optimization of batch size, number of workers, and resource allocation via system profiling contributes to more effective training dynamics, improving generalization and accuracy. As shown in Tables 2 and 3, different configurations significantly affect model performance.

3. Smoother Training Process: The semi-asynchronous mechanism adaptively controls the synchronization interval (∆T), enabling faster convergence in early stages and stable fine-tuning in later stages. This dynamic balance may indirectly enhance model performance.

4. Improved Utilization of Data and Compute Resources: Especially under resource or data heterogeneity, the proposed method enables better use of all available resources, reducing convergence variance that could negatively affect accuracy in baselines.

Q5: Given that the dynamic programming (DP) algorithm operates over a discrete and bounded search space, its computational complexity is polynomial and tractable. We give the specific analysis as follows:

• The DP algorithm in PubSub-VFL for hyperparameter optimization exhibits a polynomial time complexity, primarily determined by the size of its state space. The state is defined by a triplet $(i, j, r)$, where $i$ indexes the number of active-party workers ($w_a \in [P, Q]$), $j$ indexes passive-party workers ($w_p \in [M, N]$), and $r$ indexes feasible batch sizes $B \in \{B_1, ..., B_R\}$ with $B \leq B_{\text{max}}$. The total number of states is $(Q-P+1) \times (N-M+1) \times R$, and each state involves computing the cost function via $O(1)$ arithmetic operations (e.g., calculating forward/backward delays and communication costs). Thus, the overall complexity is $O((Q-P+1) \times (N-M+1) \times R)$, which remains manageable in practice due to constrained ranges for workers (e.g., $[2, 50]$) and batch sizes (e.g., $\{16, 32, ..., 1024\}$), ensuring efficiency for offline hyperparameter tuning. Moreover, it is a one-time offline procedure and does not run during training. Therefore, it does not introduce runtime overhead during the actual learning process.