Spiking Graph Neural Network on Riemannian Manifolds

W1: Performance on GAT backbone.

The proposed MSG is applicable to any backbone GNN, where the incoming current $x$ is given by the corresponding backbone.

We show the results of GAT backbone in the sphere manifold on Computers, Photo, CS and Physics datasets as follows. (The mean with standard derivations of 10 independent runs are reported.)

As shown in the table above, MSG achieves competitive performance on GAT and GCN backbones, and consistently outperforms the original GAT methods.

W2: Experimental settings of SNN baselines for link prediction

For the SNNs generating spiking representations (i.e., SpikeGCN and SpikeGCL), we feed the spiking representation into Fermi-Dirac decoder, obtaining the pairwise similarity for link prediction.

For the SNNs that cannot generate spiking representations (i.e., SpikeNet and SpikeGraphormer), we do not report the results of link prediction, since there exits no well-recognized similarity for the spikes.

Q1: The additional parameters and energy consumption by adding Riemannian forwarding.

The proposed MSG does not introduce additional parameters in the Riemannian forwarding.

The proposed MSG leverages the tangent vector, rather than the manifold representations, for model inference. Thus, only limited float-point operations are involved.

We report the theoretical energy consumption (mJ) in Table 3and, in fact, MSG presents competitive and even better energy efficiency to the previous Spiking GNNs, and obtain superior accuracy.

Q2: In model training, is DvM’s backward time of independent from the steps of spikes?

The gradient backpropagation is introduced in Theorem 4.1, and DvM relates to the step of spikes ONLY in Eq. (7), or more specifically, the derivate of $\frac{\partial \mathbf{v}_i}{\partial {\mathbf{x}_i[t]}}$. Note that, the pooling of Eq. (7) is the mean of $x$ sequence, and the derivative is simple and very fast to compute.

In contrast, the traditional BPTT methods recursively compute the whole backpropagation process for each time step.

Consequently, the proposed DvM achieves the significant less training time to BPTT, as shown in Fig 4.

Q3: Specify the manifold in Figure 5.

Fig. 5 is a torus, generated by rotating a circle around an axis coplanar with the circle.

The equation of the torus is specified as follows,

where $x(\theta, \phi)$, $y(\theta, \phi)$, and $z(\theta, \phi)$ represent the three-dimensional coordinates of any point on the surface of the torus.

$R$ is the radius of the major circle, which is the distance from the center of the torus to the center of the minor circle.

$r$ is the radius of the minor circle, which is the distance from the center of the minor circle to the surface of the torus.

$\theta$ is the angle parameter for points on the minor circle, used to describe the position of points on the minor circle.

$\phi$ is the angle parameter for the position of the minor circle's center on the major circle, used to describe the location of the minor circle center along the path of the major circle.

Equivalently, a torus is constructed as the product of two circles: $S^1 \times S^1$.

Q4: The difference between this paper and the recent paper (Continuous spiking graph neural networks) regarding ODE.

''Continuous spiking graph neural networks''' is the reference [11] of our paper. [11] study the connections between spiking neural network, continuous GNN, and ODE in Euclidean space, while we relate GNN and Manifold ODE, as stated in Sec. 5.

More specifically, [11] establishes ODE w.r.t. to the spiking representations, while we consider the ODE w.r.t. the manifold representations, and the spikes are related to the Charts as elaborated in Sec. 5.

W1: On Figure 4(a) and Training Time

Sorry for the typo in Figure 4(a) where we mislabeled the legend. We list the training time as follows.

The gradient backpropagation (BP) time of DvM and Surrogate on Lorentz model is shown below.

The gradient backpropagation (BP) time on Sphere model is as follows.

As shown in the tables, the proposed DvM achieves the significant less training time to BPTT, as DvM does not perform gradient backpropagation recursively for each time step.

W2: Details of model inference.

In the model inference, the proposed MSG do not need the expensive successive exponential maps, and thereby only limited float-point operations (i.e., addition) are involved.

In MSG, we leverage the tangent vectors $\mathbf{v}$’s for the downstream tasks, in accordance to the Charts of manifold ODE in Sec. 5.

Specifically, with the model parameters trained by DvM, we first conduct the pooling operation over the spikes to obtain the tangent vector as in Equation 7,

and then sum over each layer to get $\mathbf{v}_i=\epsilon\sum_l \mathbf{v}^{l-1}_i$.

Finally, $\mathbf{v}_i$ is feed in the classification head; or the Fermi-Dirac decoder for link prediction.

We will specify this point in the final version.

W1: It is not the first time the Riemann manifold has been introduced into spiking neural networks.

In this paper, the geometric concept of Riemannian manifold is a smooth manifold endowed with a Riemannian metric. To the best of our knowledge, this is the first time that Riemannian manifold is introduced into spiking neural networks. Could you please provide the relevant paper?

W2: The energy consumption is unknown for the proposed method.

The result of theoretical energy consumption is given in Table 3 where both encoding process energy and spiking process energy are calculated.

Specifically, we calculate the number of multiply-and-accumulate operations (MACs), and the number of synaptic operations operations (SOPs), following the previous works [1] [2]. The formula of theoretical energy consumption is given below,

• $ N$: the number of nodes in the graph
• $ d$: the dimension of node features
• $ L$: the number of layers in the neural model.
• $ T$: the time steps of the spikes
• $ S_l^t$: the output spikes at time step $t$ and layer $l$.

More details are given in Appendix E.3.

In addition, in the model inference, the proposed MSG leverages the tangent vectors for the downstream tasks (i.e., we sum the $\epsilon\mathbf{v}^{l-1}_i$ in Eq. (7) accumulatively for each layer.), and do not need the expensive exponential maps in the manifold. Consequently, the proposed MSG achieves low energy consumption.

[1] Li, J., H. Zhang, R. Wu, et al. A Graph is Worth 1-bit Spikes: When Graph Contrastive Learning Meets Spiking Neural Networks. arXiv preprint arXiv:2305.19306, 2023.

[2] Zhu, Z., J. Peng, J. Li, et al. Spiking graph convolutional networks. In Proceedings of the 31st IJCAI, pages 2434–2440. ijcai.org, 2022.

W3/Q2: On the real-world applications where MSG could be particularly beneficial.

First, the proposed MSG is particularly beneficial to the Large-scale Riemannian Graph Learning.

Riemannian graph models (e.g., HGCN of hyperbolic space, $\kappa$-GCN of constant curvature manifold) report superior performance in recent years. Fundamentally, Riemannian models are well aligned with the graph structures, but they are computationally expensive and thus results in high energy consumption.

Bridging this gap, MSG essentially decreases the energy consumption of Riemannian GNN, which is shown in Table 3. We list the theoretical energy consumption (mJ) of HGCN, $\kappa$-GCN and the proposed MSG here for reference.

Consequently, the proposed model is much more scalable and greener to handle large graphs.

Second, the proposed MSG is particularly beneficial to the SNN-based Link Prediction.

Existing SNNs do not achieve satisfactory results in link prediction, but MSG achieves competitive link prediction results to the state-of-the-art ANNs with the aid of Riemannian manifold, as shown in Table 1.

Q1: Details on the potential neuromorphic implementation.

The neuromorphic implementation of the proposed MSG has no difference to the previous Euclidean Spiking GNNs, such as SpikeGCL and SpikeGCN.

Specifically, referring to the model inference (the answer of W2), the proposed MSG only needs simple operations besides SNN, and do not need the complicated manifold operations, which is similar to the previous methods, such as SpikeGCL and SpikeGCN. Also, the spikes can be directly implemented on neuromorphic hardware [1][2].

[1] J. Zhao, E. Donati and G. Indiveri, "Neuromorphic Implementation of Spiking Relational Neural Network for Motor Control," 2020 2nd IEEE International Conference on Artificial Intelligence Circuits and Systems (AICAS), Genova, Italy, 2020, pp. 89-93, doi: 10.1109/AICAS48895.2020.9073829.

[2] Thiele J C, Bichler O, Dupret A. Spikegrad: An ann-equivalent computation model for implementing backpropagation with spikes[J]. arXiv preprint arXiv:1906.00851, 2019.

W1: In Table 1, some link prediction results of SNN baselines are lacked.

For the SNNs generating spiking representations (i.e., SpikeGCN and SpikeGCL), we feed the spiking representation into Fermi-Dirac decoder, obtaining the pairwise similarity for link prediction.

For the SNNs that cannot generate spiking representations (i.e., SpikeNet and SpikeGraphormer), we do not report the results of link prediction, since there exists no well-recognized similarity for the spikes. Also, SpikeNet is designed for node classification specifically.

Q1: Typo in Line 275.

Thanks for your careful review and kind reminder. Concretely, $\mathbf{y(t)}=Log_\mathbf{z}(\mathbf{z}(t))$ in the Theorem.

Q2: The difference between the sub-gradient method in reference [15] and the proposed DvM approach.

The sub-gradient method generates spike representations from corresponding spike trains, and conduct gradient backpropagation through the spike representations.

The sub-gradient method works with the Euclidean space of spike representations, while the proposed DvM conducts gradient backpropagation on the manifold.

In the sub-gradient method, the clamp operator may cause gradient error, while each term in DvM is differentiable, and thus DvM computes gradient without approximation.

Q3: Will proposed MSG suffer from gradient exploding/vanishing?

SNNs are exposed to the issue of gradient exploding/vanishing due to BPTT training, which requires recursive gradient backpropagation for each timestep. Thus, SNN tends to encounter gradient exploding/vanishing when the number of time steps is large (similar to the RNN going deeper).

In contrast, in MSG, we propose DvM training approach that conducts gradient backpropagation via the manifold and does NOT need recursive backpropagation. Moreover, from Eq. (39) in Appendix C, our formulation with the exponential map is similar to the residual connection, alleviating the gradient vanishing.

Empirically, as shown in Figure 4(a) and the results in Appendix F, the proposed MSG does not expose to gradient exploding/vanishing.

On the numerical stability of cosh and sinh. We address the issue of numerical stability in the division of $cosh$ and $sinh$, especially for the zero-value.

Specifically, we compute the limits in the form of $\frac{sinh(x)}{x}$ and $\frac{cosh(x)-1}{x^2}$ when $x\rightarrow 0$. We substitute the limits into the formulas as $x$ is less than a threshold.

Thanks for the authors' response, and the responses have addressed my concerns. Having reviewed the comments from other reviewers, I have decided to endorse this paper, and raise my score to Strong Accept.

Q: Argument, why Riemannian manifolds are required, could be strengthened, and also what about other types of manifolds?

On the one hand, Riemannian manifolds are well aligned with graph structures (e.g., the hierarchical/tree-like and cyclical structures of graphs correspond to hyperbolic and spherical manifold, respectively), and Riemannian graph models (e.g., HGCN of hyperbolic space [1], $\kappa$-GCN of constant curvature manifold [2]) report superior performance in recent years.

Note that, besides matching the graph geometry, the proposed MSG essentially decreases the energy consumption of Riemannian GNN, which is shown in Table 3. Thus, the proposed model is much more scalable and greener to handle large graphs.

On the other hand, owing to the Diffeomorphism of Riemannian manifolds, the proposed MSG obtains manifold representation based on SNN. As a result, MSG becomes a more general approach that can handle not only node classification but also link prediction.

Note that, existing SNNs do not achieve satisfactory results in link prediction, but MSG achieves competitive link prediction results to the state-of-the-art ANNs with the aid of the Riemannian manifold.

For other types of manifolds:

The diffeomorphism and distance measure of Riemannian manifold essentially support the design of DvM optimization approach and benefit link prediction, respectively.

For pseudo-Riemannian manifolds, we can apply the idea of MSG but it lacks an elegant exponential map, given the geodesic is broken in pseudo-Riemannian manifolds [3]. Thus, we consider the Riemannian manifolds, e.g., hyperbolic space, sphere, and their products.

For the topological manifolds other than Riemannian/pseudo-Riemannian ones, the alignment between graph and the manifold is not well established.

[1] Chami, I., Ying, Z., Re, C., and Leskovec, J. Hyperbolic graph convolutional neural networks. In Advances in the 32nd NeurIPS, pp. 4869–4880, 2019.

[2] Bachmann, G., G. Bécigneul, O. Ganea. Constant curvature graph convolutional networks. In Proceedings of the 37th ICML, vol. 119, pages 486–496. PMLR, 2020.

[3] Xiong, B., S. Zhu, N. Potyka, et al. Pseudo-riemannian graph convolutional networks. In Advances in NeurIPS, vol. 32. 2022.