GLASU: A Communication-Efficient Algorithm for Federated Learning with Vertically Distributed Graph Data

1. The technical solutions to the problems seem pretty straightforward and are somehow not new. The stale updating method specifically has been proposed in VFL (albeit not GNN) by (Fu et al. 2022) and (Liu et al. 2022). A method similar to the lazy aggregation method (avoid some neighborhood aggregation) is proposed in (Peng et al. 2022). Thus, although the paper indeed address an important combination of problems, the techniques are not new.

2. The problem setting seems narrow. Throughout the paper the authors seem to assume that all clients hold an identical set of nodes, and the algorithm can only operate on them, which may seem too restrictive. A set of fully aligned nodes is uncommon for different clients in VFL, and imposing such a constraint makes the setting narrow. At least, the authors should consider how to deal with mis-aligned nodes at individual clients. Also, the assumption that all clients have access to (the same set of) labels is also very strong. It is more likely that some may not have labels, or that they have distinct label spaces.

3. The experiments seem to have been set at a very simple level. First, the datasets used in this paper are simple (especially, Cora and Citeseer), as they have a limited number of nodes. Second, the 3-client setting seems small. Third, it is not clear why Sim. (i.e. simulating a centralized GNN) will lead to accuracy loss, and GLASU-1 can lead to accuracy improvements. Finally, it is also unclear why increased staleness leads to performance improvements in Table 3 (Amsterdam).

(Peng et al. 2022) Sancus: Staleness-Aware Communication-Avoiding Full-Graph Decentralized Training in Large-Scale Graph Neural Networks. VLDB 2022 (Liu et al. 2022) FedBCD: A Communication-Efficient Collaborative Learning Framework for Distributed Features. IEEE Trans on Signal Processing 2022 (Fu et al. 2022) Towards Communication-efficient Vertical Federated Learning Training via Cache-enabled Local Updates. VLDB 2022

1. The paper lacks statistical profiling results to precisely quantify the communication overhead caused by centralized training, despite the author's claim that it is significant.

2. Several concerns are raised regarding the problem setting of vertical federated learning as described in the paper: —The paper assumes that each client shares the same node set, but in real-world scenarios, it is more realistic to expect that nodes (e.g., users) from different departments only have a partial overlap. This contrasts with traditional federated learning where data from different clients can vary in sample numbers and distribution. —Privacy issue: While the authors briefly discuss the addition of differential privacy (DP) on node features, there is still a privacy concern with training labels, as the training labels need to be stored on all clients. Consider a scenario in which a department wishes to train a model using private node features from other departments. They would need to share their own private labels without any privacy guarantees.

3. GLASU employs lazy aggregation to reduce the number of synchronizations from $O(L)$ to $O(K)$. However, due to potential variations in edges across clients, this method does not address the straggler problem caused by unbalanced execution time before each synchronization.

4. Hyperparameter choice: The proposed lazy aggregation and staleness updates require the user to decide hyperparameters $K$ and $Q$. It's a question how to choose the right number as it may affect training accuracy. Moreover, how to select which layers to perform lazy aggregation is not discussed yet.

5. The experiment settings involve randomly sampling edges to form subgraphs for each client.  It would be beneficial for the authors to provide more clarification on the details of the splitting process and consider unbalanced splitting cases that resemble real-world scenarios.

6. Lack of experiment results: it's not very clear how the runtime is calculated. While the paper focuses on communication reduction, there is a lack of comparison in communication time between GLASU and baseline methods.

Regarding the "stale updates" aspect, the authors employ a method in which they store the "all but m" representation by averaging the global information after subtracting this client's specific information. While this information extraction method is straightforward, it may not perform optimally in scenarios where embeddings are identical. In such cases, the model may struggle to discern which portion of the information was omitted.

1.The authors have not addressed the privacy issue, as the node embeddings and gradient information are transmitted directly. Although they mention that privacy protection mechanisms can be applied to the proposed GLASU algorithm, no experimental verification has been provided to assess the impact of such measures. This is a crucial aspect, considering that implementing a differential privacy model may lead to a decline in accuracy, while the use of homomorphic encryption could substantially reduce efficiency. It would be helpful if the authors could conduct an analysis or experiments to demonstrate the effect of incorporating privacy protection mechanisms.

2. The proposed method heavily relies on the sampling method, as in every Q iterations, all the clients can only train the models based on the same set of sampled neighbors. Thus, it may not be easily adapted to GNNs with a global receptive field, such as graph transformers. Additionally, since the sampled set matters, different sampling strategies should be implemented to test the robustness of the stale updates.

3. According to section 3.2, all layers in {l_1,…,l_K} are aggregated. However, in Algorithm 2, the routine ends when k=2 on the server, indicating that S_m [l_1] is not aggregated. This discrepancy needs clarification.