VertiBench: Advancing Feature Distribution Diversity in Vertical Federated Learning Benchmarks

4. "Is it possible to use one metric for reflecting both importance and correlation? Or say how should a user trade-off between these two factors in comparing different VFL algorithms?"

In our revised Section 2.1 and Appendix H.1, we clarify the rationale behind choosing specific split methods depending on the scenario.

For datasets with nearly independent features, the low inter-party correlation makes correlation-based splits less meaningful, suggesting the superiority of importance-based feature splits. Conversely, in datasets with highly correlated features, assessing individual feature importance becomes impractical, making correlation-based splits more suitable due to varying inter-party correlations.

Importance and correlation are treated as orthogonal evaluation factors applicable in distinct scenarios. While there may be an intrinsic link between them, our experiments indicate that focusing on one factor at a time yields explainable results reflective of real-world performance. As discussed in Appendix H.1, the interplay between importance and correlation can be complex. A joint optimization for both factors might be computationally intensive and less explainable, while providing limited additional insights.

Essentially, splitting based on individual factors offers two key advantages:

• Efficiency: Joint optimization of multiple factors is substantially more complex and resource-intensive compared to optimizing a single factor.

• Explainability: Separate factor optimization allows for clearer insights into VFL algorithm performance. For instance, as the balance parameter $\alpha$ increases, GAL's performance tends to drop while FedOnce improves. This suggests GAL is better suited for imbalanced datasets and FedOnce for balanced ones. This inference aligns with real-world VFL dataset performances. In the relatively imbalanced Vehicle dataset (small $\alpha$), GAL outperforms FedOnce by 2%. Conversely, in the NUS-WIDE dataset, GAL's performance is comparable to FedOnce, differing by less than 0.3%.

Furthermore, although VertiBench doesn't support simultaneous factor-based splitting, it is capable of evaluating both importance and correlation effectively. This is exemplified in Figure 2, showcasing its capability in assessing both factors concurrently.

We are grateful for the insightful feedback from the reviewers and have thoroughly revised our manuscript in line with your suggestions. Below, we detail how each concern and question has been addressed. We welcome any further feedback and kindly request your reconsideration of our revised submission, including a reassessment of its score if deemed appropriate.

1. "The notation needs to be clarified."

We have fixed this issue in Equation 1. This index should be $k$ instead of $i$.

2. "Offering a more realistic VFL dataset in a related benchmark would be better"

In response to the need for a broader range of real-world VFL datasets, we have developed a new image-to-image VFL dataset, named "Satellite." This dataset fills a crucial gap, as there currently are no image-image VFL datasets available in the existing benchmarks like NUS-WIDE, Vehicle, and FedAds. The construction process and specifics of the Satellite dataset are comprehensively detailed in Appendix E. Additionally, we have conducted evaluations using SplitNN on this dataset, presented in Appendix G.4, to demonstrate its potential for improvement in VFL scenarios. This initiative not only enhances the diversity of our VFL benchmarks but also contributes a novel dataset to the field, broadening the scope for future research and application.

In this section, we outline the construction process of the Satellite dataset, which was adapted from the WorldStrat dataset~\cite{satellite}, originally intended for high-resolution imagery analysis. The Satellite dataset encompasses Point of Interest (POI) data, each associated with one or more Areas of Interest (AOI). Every AOI incorporates a unique location identifier, a land type, and 16 low-resolution, 13-channel images, each taken during a satellite visit to the location.

In our revised manuscript, we acknowledge the limitations of expanding the real scope due to dataset size constraints. To mitigate this, we provide additional insights in Section 4.4, demonstrating that VertiBench-synthetic datasets effectively represent the relative performance of VFL algorithms on real-world datasets. This approach not only broadens the applicability of our benchmarks but also offers a practical solution for evaluating VFL algorithms in scenarios where large-scale real datasets are not readily available.

3. "correlation-based split method may face scalability challenges when applied to a large number of parties, which could limit its practicality in certain scenarios."

In Appendix D.3, we provide detailed timings for feature splits across datasets, encompassing a range of up to 0.58 million instances and 20,000 features. Additionally, we conducted scalability tests for the correlation-based split method on synthetic data of varying sizes.

The importance-based method proves to be highly efficient, completing within 30 seconds across all datasets tested. This method, which randomly splits without necessitating the computation of Shapley values, still retains some Shapley-related properties, ensuring scalability for large datasets.

Conversely, the correlation-based method, while not as rapid, is sufficiently efficient for practical applications. Our scalability tests demonstrate that this method can handle up to 10,000 features, with the time cost increasing approximately linearly with the number of features. This level of efficiency is adequate for most datasets typically used in VFL, such as realsim and CIFAR10.

We emphasize that the permutation-based optimization in VertiBench is an orthogonal problem. Should more advanced optimization algorithms become available, they can be seamlessly integrated to enhance VertiBench's efficiency.

We acknowledge the misunderstanding of the concern (3) regarding the scalability of the correlation split method in relation to party size. To address this, we conducted additional experiments illustrated in Figure 10c (Appendix D.3), focusing on the impact of increasing party numbers on efficiency. The results show that within a fixed global dataset, the number of parties has minimal effect on the efficiency of the correlation split method. We trust this experiment more accurately addresses your concerns.

3. "It is confusing when reading "VertiBench ..."

We have moved this part to Appendix D.2 according to Reviewer myAZ's advice.

4. "The introduction of the two real VFL datasets is missing"

We acknowledge the earlier confusion and have now added clarifying introductions in both Section 4.2 and Appendix F for enhanced understanding.

5. "Wrong citation of Shapley-CMI"

We have corrected this wrong citation.

6. "Citation Absence or Overclaim"

In Appendix B, we have thoroughly discussed and cited all mentioned studies, including a new addition, the FedOnce algorithm, to our benchmark for deeper insights. We acknowledge the challenge of evaluating all existing VFL codes, primarily due to the unavailability of open-source versions of many algorithms. Our primary objective is to demonstrate the efficacy of VertiBench across various VFL types, rather than evaluate as many algorithms as possible. Recognizing the diverse subtypes within VFL, as indicated, we have stated this limitation in our introduction and Section 4.1.

In this paper, we concentrate on the primary types of VFL, acknowledging that there are various subtypes as identified in (Liu et al., 2022). Exploring these subtypes in depth will be an objective of our future research efforts.

We are grateful for the insightful feedback from the reviewers and have thoroughly revised our manuscript in line with your suggestions. Below, we detail how each concern and question has been addressed. We welcome any further feedback and kindly request your reconsideration of our revised submission, including a reassessment of its score if deemed appropriate.

1. "Unclear Independent Assumption" In our revised Section 2.1 and Appendix H.1, we clarify the rationale behind choosing specific split methods depending on the scenario.

For datasets with nearly independent features, the low inter-party correlation makes correlation-based splits less meaningful, suggesting the superiority of importance-based feature splits. Conversely, in datasets with highly correlated features, assessing individual feature importance becomes impractical, making correlation-based splits more suitable due to varying inter-party correlations.

Importance and correlation are treated as orthogonal evaluation factors applicable in distinct scenarios. While there may be an intrinsic link between them, our experiments indicate that focusing on one factor at a time yields explainable results reflective of real-world performance. As discussed in Appendix H.1, the interplay between importance and correlation can be complex. A joint optimization for both factors might be computationally intensive and less explainable, while providing limited additional insights.

Essentially, splitting based on individual factors offers two key advantages:

• Efficiency: Joint optimization of multiple factors is substantially more complex and resource-intensive compared to optimizing a single factor.

• Explainability: Separate factor optimization allows for clearer insights into VFL algorithm performance. For instance, as the balance parameter $\alpha$ increases, GAL's performance tends to drop while FedOnce improves. This suggests GAL is better suited for imbalanced datasets and FedOnce for balanced ones. This inference aligns with real-world VFL dataset performances. In the relatively imbalanced Vehicle dataset (small $\alpha$), GAL outperforms FedOnce by 2%. Conversely, in the NUS-WIDE dataset, GAL's performance is comparable to FedOnce, differing by less than 0.3%.

Furthermore, although VertiBench doesn't support simultaneous factor-based splitting, it is capable of evaluating both importance and correlation effectively. This is exemplified in Figure 2, showcasing its capability in assessing both factors concurrently.

2. "Seemingly Case-Restricted Significance"

To address concerns about the effect of $\alpha,\beta$, we introduced a new baseline, FedOnce, significantly influenced by split factors. For the detailed values of the performance shifts of other VFL algorithms, which may be overshadowed by C-VFL's variability, refer to Table 9. Here's a summary of how each algorithm is affected by $\alpha$ and $\beta$:

• C-VFL: Performance varies up to 40% with $\alpha$ and up to 20% with $\beta$.
• FedOnce: Shows a 10% variation with $\alpha$ on the letter dataset, and 5-7% on covtype, gisette, and epsilon; affected by 1-3% with $\beta$ on covtype and epsilon.
• GAL: Impacted by 8% with $\alpha$ on letter and radar, and 2-5% on all other datasets; minimal impact from $\beta$.
• SplitNN: Shows minimal variation due to robustness (though at a higher communication cost).
• FedTree/SecureBoost: Not affected by feature split, theoretically matching centralized training performance.

Notably, on the letter dataset, the performance ranking between FedOnce and GAL switches as $\alpha$ increases from 0.1 to 100. This suggests that GAL is better suited for imbalanced VFL scenarios, while FedOnce excels in balanced splits.

This trend aligns with real VFL datasets: For the relatively imbalanced Vehicle dataset (small $\alpha$), GAL outperforms FedOnce by 2%. In contrast, on the NUS-WIDE dataset, both GAL and FedOnce show closely matched performance (within 0.3% difference), reinforcing that these synthetic dataset conclusions can effectively mirror real-world scenarios.

We are grateful for the insightful feedback from the reviewers and have thoroughly revised our manuscript in line with your suggestions. Below, we detail how each concern and question has been addressed. We welcome any further feedback and kindly request your reconsideration of our revised submission, including a reassessment of its score if deemed appropriate.

1. "Further clarification is needed on how to interpret the findings from Figure 7 to better understand the implications of the results"

In our revised manuscript, we delve deeper into this aspect and provides novel insights in Figure 4 of Section 4.4. We specifically examined the accuracy disparities between the Vehicle and NUS-WIDE datasets, comparing these real-world discrepancies with the accuracy gaps observed in synthetic datasets split under identical $\alpha$ or $\beta$ parameters. This comparison reveals a strong positive correlation between the accuracy gaps in synthetic and real datasets. Specifically, synthetic datasets based on $\alpha$ exhibited a correlation coefficient of 0.83, while those based on $\beta$ demonstrated a coefficient of 0.39. This finding suggests that while not offering precise predictions, the performance on synthetic datasets with the same split parameters can provide a rough estimate of an algorithm's effectiveness on real VFL datasets.

2. "Explore whether Spearman rank correlation can be replaced by the other mentioned correlations and, if possible, provide a comparison of their performance."

We have include the experiments of performance split by Pearson coefficient in Appendix G.10.

In comparing Spearman rank and Pearson correlation coefficients, we find Spearman rank effective for non-linear correlations, while Pearson is suitable for linear correlations. Figure 26 illustrates the performance differences on datasets split using these metrics.

For linearly correlated datasets like gisette, Pearson's coefficient is preferable. It accurately captures the enhanced performance of C-VFL on highly correlated splits, a nuance Spearman rank-based splitting misses. Conversely, for non-linearly correlated datasets such as radar, Spearman rank-based correlation is more effective, highlighting variations in C-VFL performance that Pearson-based splits overlook. Therefore, the choice of correlation coefficient should align with the dataset's correlation characteristics.

3. "Figure 5 indicates that three out of four VFL baselines are minimally impacted by alpha and beta"

To address concerns about the effect of $\alpha,\beta$, we introduced a new baseline, FedOnce, significantly influenced by split factors. For the detailed values of the performance shifts of other VFL algorithms, which may be overshadowed by C-VFL's variability, refer to Table 9. Here's a summary of how each algorithm is affected by $\alpha$ and $\beta$:

• C-VFL: Performance varies up to 40% with $\alpha$ and up to 20% with $\beta$.
• FedOnce: Shows a 10% variation with $\alpha$ on the letter dataset, and 5-7% on covtype, gisette, and epsilon; affected by 1-3% with $\beta$ on covtype and epsilon.
• GAL: Impacted by 8% with $\alpha$ on letter and radar, and 2-5% on all other datasets; minimal impact from $\beta$.
• SplitNN: Shows minimal variation due to robustness (though at a higher communication cost).
• FedTree/SecureBoost: Not affected by feature split, theoretically matching centralized training performance.

Notably, on the letter dataset, the performance ranking between FedOnce and GAL switches as $\alpha$ increases from 0.1 to 100. This suggests that GAL is better suited for imbalanced VFL scenarios, while FedOnce excels in balanced splits.

This trend aligns with real VFL datasets: For the relatively imbalanced Vehicle dataset (small $\alpha$), GAL outperforms FedOnce by 2%. In contrast, on the NUS-WIDE dataset, both GAL and FedOnce show closely matched performance (within 0.3% difference), reinforcing that these synthetic dataset conclusions can effectively mirror real-world scenarios.

4. "Communication efficiency can be moved to Appendix"

We have relocated the original Section 4.4 to the Appendix, noting that communication efficiency in VFL is largely unaffected by variations in feature split. Specifically, if each party possesses an equal number of features, the communication cost remains consistent. Moreover, even in scenarios where the number of features differs across parties, the impact on communication cost is negligible compared to the communication cost gap arising from distinct algorithmic designs.

3. "Proposition 1: deep connections with Shapley value and feature correlation"

Proposition 1 intuitively links global data distribution with the importance and correlation among parties in machine learning models. The task of estimating this distribution, central to machine learning, poses significant challenges. Consequently, this complex relationship may not be directly translatable into mathematical terms like Shapley value or Pearson/Spearman rank correlation. Addressing the concern regarding Proposition 1's connection to Shapley values or feature correlations, it is essential to note that while the proposition is provable through mathematical manipulation, its role is more heuristic in nature. It serves as a guiding principle to understand the interplay between feature balance and correlation rather than indicating a deep, direct mathematical correlation with these concepts.

4. "Figure 2, 3, 4 can be moved to Appendix"

We have moved Figure 2,3,4 and their related contents to Appendix D.2.

5. "Communication efficiency can be moved to Appendix"

We have relocated the original Section 4.4 to the Appendix, noting that communication efficiency in VFL is largely unaffected by variations in feature split. Specifically, if each party possesses an equal number of features, the communication cost remains consistent. Moreover, even in scenarios where the number of features differs across parties, the impact on communication cost is negligible compared to the communication cost gap arising from distinct algorithmic designs.

6. "Can the authors provide further discussion of the scalability of their two partitioning methods?"

In Appendix D.3, we provide detailed timings for feature splits across datasets, encompassing a range of up to 0.58 million instances and 20,000 features. Additionally, we conducted scalability tests for the correlation-based split method on synthetic data of varying sizes.

The importance-based method proves to be highly efficient, completing within 30 seconds across all datasets tested. This method, which randomly splits without necessitating the computation of Shapley values, still retains some Shapley-related properties, ensuring scalability for large datasets.

Conversely, the correlation-based method, while not as rapid, is sufficiently efficient for practical applications. Our scalability tests demonstrate that this method can handle up to 10,000 features, with the time cost increasing approximately linearly with the number of features. This level of efficiency is adequate for most datasets typically used in VFL, such as realsim and CIFAR10.

We emphasize that the permutation-based optimization in VertiBench is an orthogonal problem. Should more advanced optimization algorithms become available, they can be seamlessly integrated to enhance VertiBench's efficiency.

We are grateful for the insightful feedback from the reviewers and have thoroughly revised our manuscript in line with your suggestions. Below, we detail how each concern and question has been addressed. We welcome any further feedback and kindly request your reconsideration of our revised submission, including a reassessment of its score if deemed appropriate.

1. "The specific form of the correlation of two party correlation Pcor"

We have refined the definition of Icor in our revision, shifting from the original average of $\text{Pcor}(\mathbf{X}_i,\mathbf{X}_j)$ to averaging the relative difference $\text{Pcor}(\mathbf{X}_i,\mathbf{X}_j) - \text{Pcor}(\mathbf{X}_i,\mathbf{X}_i)$. This new approach leads to several key advancements:

• Theoretical validation that optimizing Icor results in ideal feature splits in optimal scenarios (Theorem 1), consistent with our empirical findings on complex real datasets (see Figure 5 in Appendix D.2).
• Provision of intuitive explanations for the design of Pcor metrics.
• Visualization of the relationship between singular values, singular vectors, and inter-party correlation.
• Theoretical demonstration that the range of Pcor is within $[0,1]$.
• Theoretical proof of that Pcor and mcor (Taylor, 2020) are equivalent in measuring inner-party correlation.
• Comprehensive re-evaluation of all experiments concerning correlation-based splits.

Icor effectively measures the overall relative inter-party correlation within a global dataset. Previously, overlooking inner-party correlations of $\mathbf{X}_i$ and $\mathbf{X}_j$ led to discrepancies. High inner-party correlation typically results in increased Pcor values with other parties. The amendment to relative Pcor addresses this issue. For instance, consider two independent features $a_1, a_2$ forming two parties such that $\mathbf{X}_i=\mathbf{X}_j=[a_1,a_2]$. In this setup, $\text{Icor}(\mathbf{X}_i, \mathbf{X}_j)=0$, reflecting significant relative inter-party correlation (Icor is usually negative). Conversely, if we swap a feature to have $\mathbf{X}_i=[a_1,a_1], \mathbf{X}_j=[a_2,a_2]$, we get $\text{Pcor}(\mathbf{X}_i,\mathbf{X}_i)=1$, $\text{Pcor}(\mathbf{X}_j,\mathbf{X}_j)=0$, leading to $\text{Icor}(\mathbf{X}_i, \mathbf{X}_j)=-1$, indicative of a low inter-party correlation, aligning intuitively with expectations.

2. "Exploring the performance variations of feature partitioning under the same $\alpha$ and $\beta$."

In our analysis presented in Figure 24 and 25, we investigated the convergence of both mean and variance in algorithm performance as the number of test repetitions increased. We observe both mean and variance converges after 5-10 runs. After the convergence, there are notable differences among the algorithms: C-VFL exhibited the largest variance, whereas SplitNN and FedTree maintained low variance levels. This study highlights the significant dependency of performance variance on the specific algorithm employed.

It's important to note that variance in dataset splits is a characteristic to be leveraged, not a problem to be solved. Given that feature splits at similar importance or correlation levels are inherently non-unique, observing performance variances under identical split configurations can actually offer insights into the robustness of VFL algorithms. For instance, while SplitNN and FedTree demonstrate consistent robustness across various splits, C-VFL exhibits considerable variance in most scenarios.

To facilitate a consistent comparison among algorithms, we recommend conducting at least 5-10 repeated experiments and reporting both the mean and variance of performances. Our observations indicate that five repetitions generally yield stable mean and variance for most evaluated methods. If an algorithm displays large variance after five runs, it probably suggests potential challenges in handling real-world feature splits with similar levels of importance or correlation.

We also acknowledge and appreciate the suggestion for providing official splits to enable more consistent evaluations and rankings, recognizing its potential utility. We will provide them when releasing benchmark.