FedGMark: Certifiably Robust Watermarking for Federated Graph Learning

We thank the reviewer for appreciating the novelty of the studied problem and the proposed certified robust watermarking scheme against attacks.

W1: Clearly define the Threat Model and Problem; Who is the adversary to steal the FedGL model); Are clients and central server an adversary? Make assumptions clear

Thanks for the suggestion! See Response to Comment#1 in the global rebuttal.

W2: Privacy concerns raised by FedGMark

Response: FedGMark does not pose additional data privacy concerns. Like classic FL, all (watermarked) clients locally process their watermark data and train the model, and then submit the trained model to the server. Hence, the server cannot access the private data.

W3: Test backdoor defenses based attacks against FedGMark

See Response to Comment#4 in the global rebuttal.

W4: Introduce the "ownership verification" procedure of FedGMark

Thanks for the suggestion! See Response to Comment#2 in the global rebuttal.

We thank the reviewer for appreciating the intuition and motivation of the proposed solution and the comprehensive evaluations to support the solution.

W1: Clearly define the "Threat Model"

Thanks for the suggestion! See Response to Comment#1 in the global rebuttal.

W2: Consider black-box attacks

We emphasize this is a defense paper aiming to design a robust watermarking scheme for FedGL. As many lessons have learnt and to avoid the false sense of security [Carlini et al. 2019], an effective defense should be tested against the strongest white-box attacks. This is because a defense that is effective against (impractical) weaker attacks does not imply it is effective against stronger attacks. To this end, we assume the attacker knows all information of the target watermarked model, and this setting actually makes our defense design the most challenging. Moreover, if the defense can successfully defend against the strongest white-box attack on the watermarked FedGL model, it is naturally effective against any weaker attacks (thus including black-box attacks).

Nicholas Carlini et al., On evaluating adversarial robustness. arXiv, 2019.

W3: Replace watermark accuracy main task accuracy with robustness and fidelity.

Thank you for the suggestion. We will change the terms.

W4: Statement on “... while graphs can have varying sizes” is not entirely accurate

We agree that certain types of graphs (like wavelets) have the same size. However, our statement is for the general graph datasets whose graph sizes are varied.

We thank the reviewer for appreciating the well-motivated approach, promising performance, and robustness guarantees.

W1: Alternative key management methods

We clarify the predefined key is used by the Watermark Generator to know which local watermark is learnt for which client. It is like an identifier of the client and only needed and known by the client’s watermark generator to generate the customized watermark. This is different from the role of the key management methods in crypto. To avoid confusion, we will not use the term “private key”, but “client ID”.

W2 (Q4): Against more knowledgeable adversaries.

As suggested, we test the attacker that has access to CWG to manipulate the FedGL training. We assume some clients are malicious and test two attacks. First, we consider a passive attack where all malicious clients DO NOT use CWG to generate customized local watermarks. Model (b) in Table 5 shows the maximum WA decrease is 9%, where all clients do not use CWG.

Second, we test an active attack where malicious clients modify their watermark data’s label to obfuscate the training. Specifically, all malicious clients’ watermark data are labeled (e.g., 2) differently from the target label (e.g., 1). The results below show MA/WA is marginally affected even with 20% malicious clients.

W3 (L1): Other aggregation methods

We test M-Krum [Blanchard et al 18] and T-mean [Yin et al 19] that consider data quality (e.g., remove outlier clients). M-Krum filters a fraction $p$ of clients whose gradients largely deviated from others, while T-mean trims off a fraction $q$ of highest and lowest values for each parameter in clients’ models. We set $p=10\%$ and $q=10\%$, and the MA/WA results are as follows. We see these aggregators achieve a robustness-utility tradeoff, and MA and WA are not largely different.

Q1(L2): Scalability

See Response to Comment#3 in the global rebuttal.

Q2: (1) Deployment of FedGMark; (2) Challenges

(1) Deployment: Stage I: Server-client model for FedGMark training

• Setup: Server and clients make an agreement on, e.g., the GL model, the aggregator, and server initializes the global model.

• Local training: Clients download the global model, define their watermark data, locally train their watermarked GL model (with the CWG and RML module) using the global model, and upload the updated local model to the server;

• Server aggregation: Server aggregates local watermarked models of selected clients to update the global watermarked model.

The final global watermarked model is shared by all clients for legal use.

Stage II: Ownership verification. See Response to Comment#2 in the global rebuttal.

(2) Challenges: We consider security and privacy threats, and data quality.

• Security: How to guarantee all clients and the server are benign or detect / remove malicious ones? How to mitigate more advanced attacks?

• Privacy: Though FL methods do not access the data, they may still leak data privacy in practice. How to provably protect the data privacy, while keeping the utility?

• Data quality: Low quality data negatively affects the utility. This could reduce the interest for clients with high-quality data to participate in the FL. How to ensure all data have high quality?

Q3: Attacks use trigger reverse engineering

See Response to Comment#4 in the global rebuttal.

L3: Alternative triggers design

Response: We add some details to adjust FedGMark to the suggested alternative triggers.

To learn feature-based triggers, we first select a set of nodes from a graph as the target nodes, and learn the watermarked features for the target nodes. We use a graph $G^i = (V^i, E^i, X^i)$ from client $i$ for illustration, where $X^i$ is the node feature matrix. We then define a feature-mask $M_f^i[v_j]=1$ if $v_j \in V_w^i$ and 0 otherwise, where $V_w^i$ is the watermark node set described in the paper. Then, we introduce a feature network (FeaNet) that learns watermarked node features as $X_w^i = FeaNet(X^i) \odot M_f^i$. The FeaNet takes input $X^i$ and outputs a matrix having the same size as $X^i$, e.g., it has the same architecture as GatingNet but adjusts the input size. The corresponding watermarked graph is defined as $G_w^i=(V^i, E^i, X_w^i)$. By generating a set of watermarked graphs {$G_w^i$} for client $i$, we minimize the loss on client $i$’s both clean graphs {$G_c^i$} and {$G_w^i$}. More details about training refer to Section 3.4.

Further, to learn feature-structure triggers, we combine FeaNet (that gets $X_w^i$) with GatingNet/KeyNet (that gets $E_w^i$), and the watermarked graphs are $G_w^i=(V^i, E_w^i, X_w^i)$. We then minimize the loss on {$G_c^i$} and {$G_w^i$}.

We evaluate these triggers and the results are follows. We observe that structure information alone is sufficient to enable designing effective triggers.

L4: Performance on hyperpara.

By default, we set the learning rate (lr) to 0.01 and #local epochs (le) to 5. The results with more lr and le are below:

We see that a large lr may reduce WA, and WA slightly increases as le grows, indicating more thorough training makes our method perform better.

We thank the reviewer for appreciating the motivation and novelty of this work (first to study robust watermarking for FedGL models).

W1: Clarify the concept of ownership in FedGL

In typical FL, a server and multiple clients collaboratively train a global model stored in the server, which is used by all clients for their tasks. Accordingly, in our ownership verification problem in FedGL, all clients design their own watermark data and collaboratively train the watermarked global model, which is for joint ownership by all participating clients.

Further, since all clients have devoted computation and data to the training, they have a strong intention to jointly protect their ownership of the model. Hence, we do not consider the case where the clients did NOT participate in watermark training, but claim the ownership of the model (actually these clients do not know how to do so).

W2: (1) Global or local watermarks for ownership verification; (2) Motivation of developing local customized watermarks

(1) We use the global watermark (integration of local watermarks) for ownership verification, in line with the fact that the watermarked global model is collaboratively learnt from all clients with their local watermarks. Figure 6 in Appendix also shows global watermark is more effective than local watermarks.

(2) Learning local customized watermarks is to utilize the unique property of each client, as different clients could have different properties (e.g., distributions of their data) and their optimal watermark could be different. Also, the learnt customized watermarks enhance the ownership verification performance. For instance, Table 5 shows watermark accuracy improves 8% with local customized watermarks.

W3: Require split specific GL models to multiple submodels

Sorry for the confusion! In our design, we do not split the existing GL model into multiple submodels. Instead, we can use any GL model (e.g., a 3-layer GIN) as a submodel and each client’s GL model is an ensemble of a set of submodels (More details in Response to W5). Hence, our approach can be easily adapted to any FedGL, where it can use any aggregator or base GL model. For instance., our experiments conducted on Fed-GIN means we use the average aggregator and GIN as the submodel.

W4: Why incorporate submodels for GL

Our goal is to ensure the provable robustness of our watermarked FedGL model against layer-perturbation attacks. Recall that we propose a majority-voting based ensemble classifier on the submodels of our GL. Based on this, we can guarantee the ownership verification is provably accurate, when the #perturbed layers on GL models satisfies Eqn (1) in Thm 1.

W5: (1) What does “layer indexes” mean? (2) Structural information is important.

(1) A GL model contains multiple layers. E.g., a 8-layer GIN can be represented with layer indexes {$l_0, \ldots, l_7$}. Splitting this GIN into 4 submodels $\{GIN_1, \cdots, GIN_4\}$ with layer indexes {$l_0, l_1$} … {$l_6, l_7$} means $GIN_i$ contains layers $l_{2i}$, $l_{2i+1}$ from the GIN. However, submodels splitted in this way are coupled from each other, making them unable to defend against layer-perturbation attacks. To tackle this problem, we design the novel GL model that is an ensemble of a set of independent submodels, where each submodel is a base GL model, e.g., GIN or GCN.

(2) Yes, it is. All our submodels are GL models that take the whole graph as input, and hence retrain the graph structure information.

W6: Can this method address the potential heterogeneity issue?

Good question! Per the Response to W2, local watermarks are learnt by considering the unique properties in each client. Such unique properties may include the heterogeneity across clients’ data. To validate this, we test our method with non-IID graphs across clients, where each client holds a single label data. The MA/WA results are below:

We can see FedGMark also performs well with non-IID datasets. This implies the learnt customized watermarks indeed capture the heterogeneity of clients’ graphs.

W7: Efficiency issues

See Response to Comment#3 in the global rebuttal.

Q1: Selected clients for aggregation different from watermarked clients

In our experiments, the server randomly selects a fraction (e.g., 50%) of clients for aggregation in each training round. Also, all clients participating in the model ownership verification have watermarked data. Hence, there is no case where the set of clients selected for aggregation differs from the set of watermarked clients.

Q2: Layer-perturbation attack before or after submodel splitting?

It is before submodel splitting (ensemble).

Q3: Federally learn the local watermarking?

In our method, local watermarks are learnt using the global model (see Step 2 in Section 3.4), which is federally trained by all clients’ local models. From this point of view, local watermarks are also federally learnt by all clients.