Beyond Random Masking: When Dropout meets Graph Convolutional Networks

We greatly appreciate your thorough feedback and the time you've dedicated to reviewing our work. We believe there may have been some misinterpretations of key elements, partly due to our inadvertent errors, which could influence your assessment. We sincerely hope that our clarifications will encourage you to revisit and reassess our work. Thank you for your understanding and consideration.

W1: Clarification of Dropout within the Context of a GCN

The authors referring to the dropout and batch normalization as “our approach” in lines 409-410 and 466 is misleading since the techniques have been well-established in deep learning for improving performance. The contribution of the authors lies in the detailed theoretical analysis of these techniques within the context of a GCN. While it is a valuable contribution, the techniques should not be claimed as their approaches.

We apologize for the confusion caused by the use of the phrase "our approach." This was an inadvertent choice of wording. We have carefully revised the manuscript to remove all references to "our approach" in relation to dropout and batch normalization.

W2: Experiments of Dropout within the Context of a GCN

The major concern is with the conclusions drawn from the experiments. It is already established in deep learning that dropout and batch normalization enhance performance through regularization. Therefore, only comparing the performance in Tables 1, 2, and 3 does not provide sufficient evidence that the observed improvements are specifically due to the additional effects of dropout in graph neural networks, as analyzed in the theorems. The authors need to design experiments that can directly validate their theoretical analysis.

We acknowledge this concern and have revised our experimental section to better connect the theoretical analysis with the experimental findings. While Tables 1, 2, and 3 were originally intended to show that dropout’s effects in GNNs have been underexplored, we now include experiments that directly validate key insights from the theorems.

Specifically, we already conducted experiments to demonstrate the formation of dimension-specific stochastic subgraphs induced by dropout (see Fig. 1-3 in Section 3.2), its degree-dependent effects (see Fig. 4-6 in Section 3.3), and its role in mitigating oversmoothing (see Fig. 7-8 in Section 3.4 and Section 3.6).

Additionally, we have incorporated some new results into the updated manuscript (see Fig. 9-10), which are also provided below in our response to W3 & W4.

W5: Minor Issues

Repeated use of variable 'd' for denoting degree (line 133) and node feature dimensionality (line 141).

Thank you for checking on that level of detail! We have updated the paper accordingly.

Q1: Exploration of Dimension-Specific Subgraphs

Section 3.2 introduces the concept of dimension-specific subgraphs. How does this impact the aggregated node representation (including all dimensions)? Since the performance of a GCN ultimately depends on the aggregated representation, it would be insightful to explore this relationship.

To address this question, we analyze different dropping methods through their masking mechanisms, subgraph formations, and degree-dependent effects, which collectively influence how features are learned and aggregated across dimensions. Our analysis reveals why dimension-specific methods, particularly dropout, lead to more robust and discriminative aggregated representations.

Different Masking Mechanisms and Their Impact on Aggregation

DropNode:

$\mathbf{M}\_d = \tilde{\mathbf{A}}((\mathbf{M}\_{node} \odot \mathbf{H}^{(l-1)})\mathbf{W}^{(l)})\_d,$ where $\mathbf{M}\_{node}$ is shared across dimensions.

DropEdge:

$\mathbf{M}\_d = (\mathbf{M}\_{edge} \odot \tilde{\mathbf{A}})(\mathbf{H}^{(l-1)}\mathbf{W}^{(l)})\_d,$ where $\mathbf{M}\_{edge}$ creates same edge mask for all dimensions.

DropMessage:

$\mathbf{M}\_d = \tilde{\mathbf{A}}(\mathbf{M}\_{msg\_d} \odot(\mathbf{H}^{(l-1)}\mathbf{W}^{(l)}))\_d,$ where $\mathbf{M}\_{msg\_d}$ varies per dimension.

Dropout:

$\mathbf{M}\_d = \mathbf{M}\_{feat\_d} \odot \tilde{\mathbf{A}}(\mathbf{H}^{(l-1)}\mathbf{W}^{(l)})\_d,$ where $\mathbf{M}\_{feat\_d}$ is dimension-specific.

First, DropNode and DropEdge apply a uniform mask across all dimensions, which limits the diversity of feature combinations and may result in homogenized representations. Second, DropMessage enhances message passing diversity by allowing dimension-specific message dropping with $\mathbf{M}\_{msg\_d}$, enabling different features to take unique paths across dimensions. Similarly, Dropout employs dimension-specific feature masking using $\mathbf{M}\_{feat\_d}$, which allows features in each dimension to learn from distinct node subsets, thereby creating varied feature combinations.

Analysis of Subgraph Formation and Degree-Dependent Effects

DropNode:

$\mathcal{G}\_t = (\mathcal{V}\setminus\mathcal{V}\_{dropped}, \mathcal{E}\setminus{(i,j)|i\in\mathcal{V}\_{dropped} \text{ or } j\in\mathcal{V}\_{dropped}}), \mathcal{V}\_{dropped} = \\{ i | \mathbf{M}\_{node\_i} = 0 \\},$

$\mathbb{E}[deg\_i^{eff}(t)] = deg\_i \prod\_{j\in\mathcal{N}(i)}(1-p)$

DropEdge:

$\mathcal{G}\_t = (\mathcal{V}, \mathcal{E}\setminus\mathcal{E}\_{dropped}), \mathcal{E}\_{dropped} = \\{(i,j) | \mathbf{M}\_{edge\_{ij}} = 0\\},$

$\mathbb{E}[deg\_i^{eff}(t)] = (1-p)deg\_i$

DropMessage:

$\mathcal{G}\_t^d = (\mathcal{V}, \mathcal{E}\_t^d), \mathcal{E}\_t^d = \\{(i,j) \in \mathcal{E} | \mathbf{M}\_{msg\_{d\_{ij}}} \neq 0\\},$

$\mathbb{E}[deg\_i^{eff}(t)] = (1-p)deg\_i$

Dropout:

$\mathcal{G}\_t^d = (\mathcal{V}, \mathcal{E}\_t^d), \mathcal{E}\_t^d = \\{(i,j) \in \mathcal{E} | \mathbf{M}\_{feat\_{d\_i}} \neq 0 \text{ and } \mathbf{M}\_{feat\_{d\_j}} \neq 0\\},$

$\mathbb{E}[deg\_i^{eff}(t)] = (1-p)^2deg\_i$

The analysis above reveals several key advantages of dropout over other methods in GNNs. First, dropout creates dimension-specific subgraphs $\mathcal{G}\_t^d$ through its feature-level masking mechanism $\mathbf{M}\_{feat\_d}$, allowing different dimensions to learn from diverse structural patterns. Second, its quadratic effect on effective degree $\mathbb{E}[deg\_i^{eff}(t)] = (1-p)^2deg\_i$ provides stronger and more adaptive regularization, particularly at high-degree nodes where oversmoothing is most problematic. This is particularly crucial as these hub nodes tend to dominate message passing and accelerate feature homogenization. These properties together make dropout especially effective at balancing feature propagation and regularization, resulting in more robust and discriminative aggregated representations compared to methods that either lack dimension-specific patterns (DropNode, DropEdge) or provide insufficient control at critical high-degree nodes (DropMessage).

W3 & W4: Experimental Evidence on Over-Smoothing

Line 472 (regarding Table 1) and line 483 (regarding Table 2) draw contrasting conclusions about the effect of dropout on Dirichlet Energy. What is the reason behind this difference in the behavior of dropout?

We apologize for the confusion. We discovered that the Dirichlet energy values reported for the graph classification experiments in Tables 2 and 3 were mistakenly copied incorrectly. The corrected results are consistent with the findings in Table 1 and support the interpretation that dropout increases Dirichlet energy. The updated values are as follows:

The results demonstrate that dropout consistently increases Dirichlet energy across all datasets, aligning with our theoretical analysis in Section 3.4. The increase ranges from approximately 48% (Peptides-func) to 141% (Peptides-struct), supporting our hypothesis that dropout helps preserve feature distinctiveness.

The code and log files can be accessed via the anonymous link in our paper. We appreciate your understanding.

Section 3.4 describes an interesting connection between dropout, the number of GCN layers, and over-smoothing. However, the authors fail to provide experimental evidence to support this relationship. Demonstrating how dropout affects over-smoothing in GCN with varying layer depths would strengthen the paper.

Thank you for the insightful comments. Firstly, Theorem 15 elucidates how dropout influences feature energy. From a static perspective, dropout increases the upper bound of feature energy. Dynamically, the effect of dropout on feature energy is co-determined by network depth, graph structure, and weight properties. To further validate how dropout affects oversmoothing, we conducted extensive experiments examining both weight matrices and feature representations. Our results demonstrate that dropout effectively mitigates oversmoothing through two key mechanisms: gradient amplification and enhanced feature discrimination. The updates are presented in two parts: weight matrix analysis and feature representation analysis.

1. Weight Matrix Analysis

We investigated how dropout affects the weight matrices in GCNs, focusing on its influence on the spectral norm $\|\mathbf{W}\|_2^2$, as detailed in Appendix A.5. Dropout introduces a scaling factor of $1/(1−p)$ in the gradients during backpropagation, leading to the following effects:

• Gradient Amplification: For surviving features (where the dropout mask is active), the gradients are amplified by $1/(1−p)$, resulting in larger weight updates during training.
• Accumulated Effects Over Iterations: Over multiple training iterations, this consistent amplification accumulates, systematically increasing the spectral norms of the weight matrices.

Empirical results, presented in Figure 10, confirm a clear correlation between dropout rates and the spectral norm $\|\mathbf{W}\|_2^2$. Larger spectral norms contribute to higher feature energy during inference. This increase in feature energy help mitigate mitigate oversmoothing. This increased spectral norm directly supports our theoretical prediction in Section 3.4 that dropout helps maintain distinct node representations across deeper layers.

2. Feature Representation Analysis

To evaluate how dropout affects feature representations, we further analyzed three key metrics, as shown in Figure 9. These experiments demonstrate the nuanced impact of dropout on mitigating oversmoothing:

• Frobenius Norm: The Frobenius norm of the feature matrix remains relatively stable with or without dropout, indicating that dropout’s effect is not a simple uniform scaling of all features.
• Pairwise Node Distances: Dropout consistently doubles the average pairwise distance between nodes, helping maintain more distinctive node representations.
• Dirichlet Energy: Dropout leads to a substantial increase in Dirichlet energy, much greater than the changes in Frobenius norm and pairwise distances. This rise in Dirichlet energy enhances the discriminative power between connected nodes, directly explaining its effectiveness in preventing oversmoothing.

The updated results and analyses provide a unified understanding of how dropout influences both feature representations and weight matrices to mitigate oversmoothing.

We sincerely thank the reviewer for raising this critical question, which addresses the core motivation of our work: understanding whether the primary mechanism of dropout in GCNs is mitigating oversmoothing rather than the traditional view of preventing co-adaptation of neurons.

Our analysis provides compelling evidence that dropout in GCNs primarily mitigates oversmoothing rather than just preventing co-adaptation. Here's why:

If dropout merely prevented co-adaptation (as in standard neural networks), we would expect it to affect all feature relationships similarly. However, our experimental results tell a different story.

We report two sets of experiments. In the first set, we used a fixed GCN architecture for all datasets. In the second set, we used the optimized GCN architecture for each dataset individually, as in Table 1 of our manuscript.

We highlight three key observations:

First, the Frobenius norm of features remains stable with and without dropout. This stability is crucial - it tells us that dropout isn't simply scaling all features uniformly, but rather selectively affecting feature relationships.

Second, when we examine the average pairwise distance between all nodes (including unconnected pairs), we see it doubles with dropout. This shows a general increase in feature distinctiveness across the graph.

However - and this is the key insight - when we look at the Dirichlet energy (distances between connected nodes), we observe a significantly more dramatic increase. This disproportionate effect on connected nodes raises an important question: if dropout were only preventing co-adaptation, why would it have a stronger effect on connected nodes compared to unconnected ones?

This disparity reveals dropout's specific role in GCNs: it intensifies the differentiation of features between connected nodes - precisely where oversmoothing tends to make features overly similar. The observation that connected nodes show stronger feature separation than the general node population clearly demonstrates that dropout actively counteracts the oversmoothing process.

This targeted impact on connected nodes, combined with the preservation of overall feature magnitudes (stable Frobenius norm), provides strong evidence that dropout's primary role in GCNs is mitigating oversmoothing, extending beyond its traditional role of preventing co-adaptation.

Thank you for the detailed results.

To further illustrate the fundamental differences in the effects of dropout between MLPs and GNNs, we are pleased to share that our new analysis of the generalization bound (as mentioned in our reply to Reviewer UjFe) suggests that graph connectivity offers natural regularization.

I like this additional effect of dropout on the generalization bound when applied to GCNs.

However - and this is the key insight - when we look at the Dirichlet energy (distances between connected nodes), we observe a significantly more dramatic increase. This disproportionate effect on connected nodes raises an important question: if dropout were only preventing co-adaptation, why would it have a stronger effect on connected nodes compared to unconnected ones?

The analysis is reasonable but does not convincingly address my concern.

Further, we conducted an experiment using both MLP and GNN on node features, with and without dropout. These results demonstrate that GNNs benefit more from dropout, supporting a coherent narrative:

Thank you for these results. The results demonstrate that dropout applied to GCNs yields a greater performance improvement than MLPs (especially for Citeseer and Cora graphs), supporting the Dirichlet energy analysis referenced in the above point.

Considering everything we discussed, I have raised my rating for this paper.

We greatly appreciate your detailed feedback. We hope our response below effectively addresses your concerns.

W1: Detailed Experimental Information

The experimental validation lacks detailed information about the 16 datasets used, and the comparative analysis with state-of-the-art methods could be more comprehensive. Some experimental results mentioned in figures are truncated in the provided content.

Thank you for your constructive feedback. We have now updated the paper to include a thorough description of the datasets, as well as the experimental results and a more comprehensive comparative analysis with state-of-the-art methods in the revised PDF. We hope these revisions address your concerns.

W2: Theoretical Assumptions and Extensions

The theoretical framework makes limiting assumptions about undirected graphs, and doesn't adequately address the extension to directed graphs. The interaction between dropout and different activation functions, as well as the impact of graph density on dropout effectiveness, need more exploration.

We appreciate the reviewer’s insightful comment on theoretical extensions. In response, we have extended our framework in three key directions:

1. Directed Graph Extension

We are grateful for the reviewer’s observation regarding directed graphs. Our framework can be naturally extended to directed graphs through a bipartite graph transformation that preserves all theoretical properties while capturing directional information flow. Specifically, any directed graph with adjacency matrix $\mathbf{A}$ can be transformed into an undirected bipartite graph:

$\tilde{\mathbf{A}}_{bipartite} = \begin{bmatrix} \mathbf{0} & \mathbf{A} ; \mathbf{A}^T & \mathbf{0} \end{bmatrix}$

This transformation establishes a one-to-one correspondence between directed graphs and their bipartite representations: each node in the original graph splits into an "in-node" and an "out-node", and each directed edge maps to an undirected edge between corresponding in/out nodes. This preserves the directed information flow while allowing us to apply our undirected graph theory.

Our key theorems extend naturally to this bipartite representation, revealing several interesting properties unique to directed graphs:

1. Feature Propagation in Directed Graphs

   Theorem 1 (Two-Step Feature Flow). For directed message passing with dropout:

   $\mathbf{H}\_{in}^{(l+1)} = \sigma(\mathbf{A}\mathbf{M}\_{out}^{(l)}\mathbf{H}\_{out}^{(l)})$

   $\mathbf{H}\_{out}^{(l+1)} = \sigma(\mathbf{A}^T\mathbf{M}\_{in}^{(l)}\mathbf{H}\_{in}^{(l)})$

   Unlike undirected graphs where information flows freely between nodes, the bipartite structure enforces alternating in/out updates, providing clearer separation of directional information flow and more controlled feature evolution.

2. Dimension-Specific Sub-graphs in Directed Setting

   Theorem 2 (Directed Sub-graph Diversity).

   $\mathbb{E}[|{G\_t^{(l,d)}}|] = d\_l(1-(1-p)^{2|\mathcal{E}|}) \cdot \frac{|\mathcal{E}\_{in}| \cdot |\mathcal{E}\_{out}|}{|\mathcal{E}|^2}$

   The sub-graph diversity now incorporates the balance between incoming and outgoing edges, a property absent in undirected graphs, leading to more nuanced structural sampling.

3. Directional Degree-Dependent Effects

   Theorem 3 (Directed Degree Effects). For node $i$ with in-degree $deg\_i^{in}$ and out-degree $deg\_i^{out}$:

   $\mathbb{E}[deg\_{i,total}^{eff}(t)] = (1-p)^4(deg\_i^{in}deg\_i^{out})^{1/2}$

   While undirected graphs have a simple quadratic dropout effect, directed graphs exhibit a fourth-power effect due to the two-step nature of information flow, providing stronger regularization for highly connected nodes.

4. Spectral Properties of Directed Dropout

   Theorem 4 (Directed Spectral Effects).

   $\lambda(\tilde{\mathbf{A}}\_{bipartite}) = {\pm\sqrt{\lambda(\mathbf{A}\mathbf{A}^T)}},$ where $\lambda$ denotes eigenvalues.

   The bipartite transformation yields symmetric eigenvalue distribution, unlike the potentially asymmetric spectrum of directed graphs, providing better-behaved gradient flow properties.

2. Different Activation Functions

Regarding the activation functions, our theoretical analysis naturally extends to different activation functions as long as they satisfy the Lipschitz continuity condition. Specifically, let $\sigma$ be any activation function that is Lipschitz continuous with constant $L\_s$, i.e.:

$|\sigma(x) - \sigma(y)| \leq L\_s|x - y|$

This condition is satisfied by most common activation functions in deep learning, including:

• ReLU: $L\_s = 1$
• Sigmoid: $L\_s = 1/4$
• Tanh: $L\_s = 1$
• LeakyReLU: $L\_s = \max(1, \alpha)$ where $\alpha$ is the slope parameter

Under this condition, our key theorems hold with only a constant factor difference. For example, in the oversmoothing analysis:

$\mathbb{E}[E(\mathbf{H}^{(l)})] \leq \frac{deg\_{\max}}{|\mathcal{E}|}(\frac{L\_s}{1-p})^{l}||\mathbf{\tilde{A}}||\_2^{2l}\prod\_{i=1}^{l}||\mathbf{W}^{(i)}||\_2^2||\mathbf{X}||\_F^2$

The Lipschitz condition ensures bounded gradient propagation, stable feature transformation, and controlled error accumulation.

Therefore, our analysis is not limited to ReLU but applies to a broad class of activation functions, making the framework more general and practical.

3. Graph Density Impact on Dropout

For a graph with density $\rho = \frac{2|\mathcal{E}|}{|\mathcal{V}|(|\mathcal{V}|-1)}$, the number of messages per feature dimension is proportional to $|\mathcal{E}|$. With dropout rate $p$, the expected number of active messages is given by:

$\text{Expected Active Messages} = (1-p)|\mathcal{E}| = (1-p)\frac{\rho|\mathcal{V}|(|\mathcal{V}|-1)}{2}$

Dense graphs (high $\rho$) have more messages to drop, as increasing density (e.g., from 0.01 to 0.1 in a 1000-node graph) raises expected active messages from 5,000 to 50,000, while also requiring careful dropout rate selection, as even small changes can significantly reduce active messages.

For a node $i$ with degree $d\_i$, the effective degree is:

$\mathbb{E}[deg\_i^{eff}(t)] = (1-p)^2deg\_i$

This indicates that higher-degree nodes experience stronger degree-dependent effects, such as a hub node with $d\_i$ = 100 being reduced to 25 when $p$ = 0.5, compared to a peripheral node with $d\_i$ = 5 being reduced to 1.25.

The expected size of a sampled subgraph is:

$\mathbb{E}[|{\mathcal{G}\_t^{(l,j)} | j=1,...,d\_l}|] = d\_l(1-(1-p)^{2|\mathcal{E}|})$

As graph density increases, this metric grows, enabling richer structural variations during training and facilitating better exploration of graph patterns. Higher density allows more distinct subgraphs to be generated per dropout iteration, particularly in dense regions, which enhances the model's ability to learn diverse structural representations.

W3: Dropout Rates and Computational Overhead

The paper lacks clear guidelines for selecting optimal dropout rates.

We appreciate your insightful comment!

Our theoretical framework provides upper bounds on the energy in an $l$-layer GCN:

$\mathbb{E}[E(\mathbf{H}^{(l)})] \leq \frac{deg\_{\max}}{|\mathcal{E}|}(\frac{L\_s}{1-p})^{l}||\mathbf{\tilde{A}}||\_2^{2l}\prod\_{i=1}^{l}||\mathbf{W}^{(i)}||\_2^2||\mathbf{X}||\_F^2$

Specifically, the graph's structural properties, particularly sparsity (captured by $deg\_{\max}/|\mathcal{E}|$), influence the effectiveness of dropout. Sparse graphs benefit more from adaptive dropout adjustments.

Furthermore, the factor $\left( \frac{1}{1-p} \right)^{l}$ underscores the role of dropout in preserving energy as the network depth increases. Increasing $p$ in deeper layers may mitigate oversmoothing, as also demonstrated in the Practical Implications of Generalization Bounds (see reply to UjFe, W1).

The paper lacks analysis of scalability to very large graphs, and discussion of computational overhead for implementing the theoretical framework. How does the computational overhead compare to traditional dropout implementations?

In Equation 4, dropout involves randomly zeroing out some of the features during training. This process typically consists of two steps: generating a mask for each feature and applying this mask. The mask is applied to the feature matrix via element-wise multiplication, which has a time complexity of $O(|\mathcal{V}| \times D$), where $|\mathcal{V}|$ is the number of nodes and $D$ is the number of features per node. Despite the introduction of dropout, the overall time complexity remains equivalent to scenarios without dropout, which is $O(|\mathcal{E}|D)$ where $|\mathcal{E}|$ is the number of edges. This equivalence arises because the primary computational burden still originates from the matrix multiplications in GNN layers. Thus, integrating dropout does not alter the fundamental computational complexity of the operations.

We greatly appreciate the very detailed feedback and your recognition of our contributions! We hope our response below will further enhance your confidence in our work.

W1 & Q1: Dropout Rates

Dropout is a general and well-known technique, to achieve performance gain via dropout, the question can be how to tune the parameter. Can the theoretical analysis of dropout in GCNs provide insights on how to select the dropout hyperparameter?

Can the theoretical analysis of dropout in GCNs provide insights on how to select the dropout hyperparameter?

Thank you for the thoughtful comment!

Our theoretical framework provides upper bounds on the energy in an $l$-layer GCN:

$\mathbb{E}[E(\mathbf{H}^{(l)})] \leq \frac{deg\_{\max}}{|\mathcal{E}|}(\frac{L\_s}{1-p})^{l}||\mathbf{\tilde{A}}||\_2^{2l}\prod\_{i=1}^{l}||\mathbf{W}^{(i)}||\_2^2||\mathbf{X}||\_F^2$

Specifically, the graph's structural properties, particularly sparsity (captured by $deg\_{\max}/|\mathcal{E}|$), influence the effectiveness of dropout. Sparse graphs benefit more from adaptive dropout adjustments.

Furthermore, the factor $\left( \frac{1}{1-p} \right)^{l}$ underscores the role of dropout in preserving energy as the network depth increases. Increasing $p$ in deeper layers may mitigate oversmoothing, as also demonstrated in the Practical Implications of Generalization Bounds (see reply to UjFe, W1).

W2: Comparison with Other Graph Learning Regularization Techniques

The paper primarily focuses on dropout in GCNs, but it may not sufficiently compare the method with other graph learning regularization techniques.

This is a fair point. Following your suggestions, we conducted experiments on additional graph learning regularization techniques [1,2,3,4], adhering to our hyperparameter settings across the Cora, Citeseer, and Pubmed datasets. The summarized results in the table below reveal that although various regularization strategies achieve similar performance levels, traditional dropout, as implemented in our study, usually performs best.

[1] DropEdge: Towards Deep Graph Convolutional Networks on Node Classification.

[2] Graph Random Neural Network for Semi-Supervised Learning on Graphs.

[3] DropMessage: Unifying Random Dropping for Graph Neural Networks.

[4] Rethinking Graph Regularization for Graph Neural Networks.

We greatly appreciate your valuable feedback and have just posted our response to your comments. We noticed your rating dropped from 6 to 5 before our response was posted, and we apologize for the delay. We hope our response adequately addresses your points and restores your confidence in our work. We look forward to hearing your thoughts.

W1: Practical Implications of Generalization Bounds

The authors provide generalization bounds for graph neural networks with dropout. However, further clarification is needed on how this finding offers insights into understanding and designing graph neural networks, or any specific guidance on selecting dropout rates. This would help demonstrate the practical relevance of the theory. Additionally, can the experiments provide corresponding analyses regarding this theory?

Thanks for your insightful suggestion. The answer is indeed yes.

Our theoretical analysis primarily aimed to illustrate how dropout impacts the robustness of graph neural networks by controlling the variance in node feature propagation across layers. The derived generalization bound:

$\mathbb{E}\_D[L(F(x))] - \mathbb{E}\_S[L(F(x))] \leq O\left(\sqrt{\frac{\ln(1/\delta)}{n}}\right)\sum_{l=1}^L L\_{loss} \cdot L\_l \cdot \sqrt{\frac{p\_l}{1-p\_l}}\|\sigma(\tilde{\mathbf{A}} \mathbf{H}^{(l-1)} \mathbf{W}^{(l)})\|\_F,$

$L\_l = \prod\_{i=l+1}^L (\|\mathbf{W}^{(i)}\|\cdot\|\tilde{\mathbf{A}}\|)$

highlights the role of dropout rates $p\_l$ in balancing the expressivity and stability of model training. This equation suggests that by carefully adjusting $p\_l$, practitioners can mitigate the risk of overfitting while ensuring sufficient model complexity for learning from graph-structured data.

In our analysis of a 3-layer GCN model, we observed that $L\_l$ decreases as depth increases. This shows that feature propagation becomes less complex in deeper layers, leading to a reduction in the model’s capacity to represent the data. Specifically, we have the following for each layer:

• Layer 1: $L\_1 = (\|\mathbf{W}^{(2)}\|\cdot\|\tilde{\mathbf{A}}\|)(\|\mathbf{W}^{(3)}\|\cdot\|\tilde{\mathbf{A}}\|)$
• Layer 2: $L\_2 = \|\mathbf{W}^{(3)}\|\cdot\|\tilde{\mathbf{A}}\|$
• Layer 3: $L\_3 = 1$

This analysis clearly shows that as the number of product terms decreases, $L\_l$ becomes smaller in deeper layers. This indicates that the complexity of feature propagation reduces, which suggests that we need to increase the dropout rate to compensate for this effect in deeper layers.

To address this, we propose a layer-specific adjustment of dropout rates to counteract the depth-dependent decrease in $L_l$. Specifically, we use a linearly increasing dropout scheme:

• Dropout rate: $p\_l = \text{initial rate} + 0.1 \times i$ (with $i$ being the index of the GNN layer).

To substantiate our theoretical insights, we conducted experiments on SAGE, implementing a linearly increasing dropout rate across each layer, where the dropout rate for any given layer is defined by the formula above. We varied the initial rates from 0.1 to 0.7 in our study. The summarized results are shown in the table below, indicating that this approach leads to performance improvements: