SGHormerVQ: Bridging Graph Transformers and Spiking Neural Networks via Spiking Vector Quantization

(Q1): The model performs less effectively on datasets with high node degrees (e.g., ogbn-products), suggesting that SVQ may suffer from information loss in highly connected graphs.

R1: In this work, we focus on analyzing the role of spiking neurons in prior spiking graph neural networks from the perspective of vector quantization. To this end, in SVQ, we just adopted the plain spiking neuron, message propagation algorithm, and normalization layer, as shown in Eq.(10)–(12). And experiments demonstrate that SGHormerVQ composed of simple components has achieved significant improvements in both efficiency and predictive accuracy for most cases. Recent studies have proposed more sophisticated alternatives to the above modules. We believe that employing these advanced alternatives with lower quantization errors can effectively address the limitation.

(Q2): The practical impact of SVQ and CGSA on computational complexity, hardware requirements, and performance trade-offs is not sufficiently analyzed, potentially limiting the model's real-world applicability.

R2: Thank you for your insightful suggestions. We update Appendix B to better illustrate the model's real-world applicability. The energy efficiency analysis is conducted based on three metrics (Memory usage, theoretical energy consumption and inference latency) for showcasing the computational complexity and performance trade-offs of SGHormerVQ. Additionally, we detail the theoretical energy consumption in the formula and discuss the deployment of the model on neuromorphic hardware. The experimental results are as follows:

Q3: I would also recommend the authors further break down the SVQ module to explore the effects of specific components (e.g., varying quantization levels, spike neuron types, or normalization techniques) could highlight why SVQ is particularly effective and reveal which configurations yield optimal performance.

R5: We have updated more ablation experiments to explore the effects of specific components (spike neuron types and normalization techniques) in section 5.4. We implement two common spike neurons (IF and LIF), and two normalization algorithms (LayerNorm and STFNorm [6]) to construct 4 baselines. The results are as follows. It shows that PLIF models, which have learnable membrane time constants and synaptic weights, achieve slightly better performance in most cases. The well-designed normalization algorithm for spiking neurons, STFNorm, outperforms the LayerNorm algorithm across all datasets.

Q4: I am curious - does the degree of homophily play a role in the SVQ?

R6: We discuss SGHormerVQ in the perspective of homophily in Appendix D. The same graph generation strategy is implemented on Cora and Citeseer datasets following the previous study [7]. The following results show the influences of different homophily ratios on predictive results, where the homophily ratio is defined as $\frac{|\{(v,w):(v,w)\in\mathcal{E}\cap y_v =y_w\}|}{|\mathcal{E}|}$. We find that as the homophily ratio decreases, the classification performance initially declines but eventually starts to improve. It implies that SGHormerVQ may achieve strong performances on certain heterophilic graphs.

[1] Fois A, et al. A Spiking Neural Architecture for Vector Quantization and Clustering. ICONIP, 2020.

[2] Liu M, et al. Spiking-diffusion: Vector quantized discrete diffusion model with spiking neural networks. arXiv, 2023.

[3] Chen J, et al. NAGphormer: A tokenized graph transformer for node classification in large graphs. ICLR, 2023.

[4] Wu Q, et al. SGFormer: Single-Layer Graph Transformers with Approximation-Free Linear Complexity. NIPS, 2023.

[5] Yao M et al. Spike-driven transformer v2: Meta spiking neural network architecture inspiring the design of next-generation neuromorphic chips. ICLR, 2024.

[6] Xu M et al. Exploiting spiking dynamics with spatial-temporal feature normalization in graph learning. IJCAI, 2021.

[7] Ma Y et al. Is homophily a necessity for graph neural networks?. ICLR, 2022.

(Q1): I am confused by the statement "SGHormerVQ with better performance infers faster than other GTs by up to 518x" as the Fig 4 shows that it has higher latency than the standard GAT with slightly higher accuracy? Also, the authors only show this for the simplest dataset (Physics dataset) - I would recommend the authors to also show the latency results for the more complex datasets. Also, how are you getting the latency numbers? Is it just the average for all the inferences in the test set?

R3: Thank you for your helpful advice. Compared to GOAT, SGHormerVQ with better performance infers faster than the VQ-based GT by 518x on the Physics dataset. We apologize for the confusion and have revised this unclear description. In the implementation, we record the absolute elapsed running time per test epoch as inference latency for SGHormerVQ and other GT baselines. Following the same settings in the previous study [4], we use the mini-batch partition for training on the ogbn-products dataset. Corresponding experiments about memory usage, inference latency and theoretical energy consumption are provided in Appendix B. Here, we present inference latency (s) results on CS, Physics, arXiv and Products datasets. The results show that the inference latency of SGHormerVQ is comparable to or even better than SGFormer.

(Q2): I would recommend the authors to give a detailed analysis of energy and memory usage would give solid proof of the paper's efficiency claims. The authors could measure the average number of spikes per inference and that would be a good measure for the memory reductions, especially compared to standard Graph Transformers and SNN-based models.

R4: Thank you for the suggestions. We provide a detailed analysis of energy and memory usage in Appendix B. The theoretical energy consumption is calculated by measuring the firing rate [5]. Furthermore, we detail the theoretical energy consumption of SGHormerVQ in the formulation. Energy and memory usage are as follows. It shows that SGHormerVQ with better performances bringing a slight extra energy cost compared to pure SNN-based GT, SpikeGraphormer.

We thank the reviewer for reading our paper.

(W1:) I am not sure what is the key novelty of the work - it seems the authors have just used Spiking Vector Quantization (which is itself not a new concept) for spiking graph transformers (also not new)

R1: Firstly, We would like to emphasize the distinction between our current method and previous approaches that involve the concept of spiking vector quantization [1][2]. In these studies, constructing or generating the codebook does not involve any strong inductive biases and the problem of Codebook Collapse is rarely discussed. We designed SGHormerVQ to address these issues. The SVQ module actively captures the dynamic during the message propagation to generate a codebook with graph inductive bias. Compared with methods based on pre-defined codebooks, SVQ can be seen as finding the optimal mapping function from nodes to codewords.

Most spiking graph transformers tend to consider spiking neurons as low-power units. The role of spiking neurons in spiking graph neural networks has been insufficiently explored. SGHormerVQ takes it a step further. In this work, we summarize and compare the strengths and weaknesses of previous SGNN designs (Appendix A). It suggests that spiking neurons not only are the foundation of low-power models but also capture the dynamics during message propagation. The architecture we proposed achieves efficiency improvement through spiking vector quantization, while indirectly incorporating graph structural information into the Transformer in the form of codewords to improve the predictive performances.

(W2), (W3): "The experimental section is very weak as most of the datasets used for the results are very simple datasets.", "The paper gives a lot of context and discusses a lot about the prior works.i would recommend to push them to the supplementary section and add more experimental results to show where this model actually works and where it fails- as that is not really clear from the experiments"

R2: Thanks for the suggestions. In experiments, most datasets are consistent with previous studies [3][4]. And the primary goal of our experiments is to verify the effectiveness of the proposed spiking vector quantization and evaluate the energy efficiency of CGSA. We update a comprehensive energy efficiency analysis in Appendix B and evaluate our methods on two heterophilic datasets (Actor and Deezer) in Appendix D. Besides, we have updated more explanations and descriptions of the experiments in the related sections.

I would like to thank the authors for their response and have raised my rating. This is a very interesting paper - however I have some more clarifications that would be great if the authors could provide:

1.It seems from the ablation studies, the use of STFNorm is even better than what the authors are using - can the authors give more insights what they believe is happening and why we are seeing this?

2. Personally I found your discussion in Appendix A to be very interesting, but too short - would it be possible for the authors to add more details of the comparisons and how they are interpreting the visualizations etc.

3. I am not very convinced with the empirical results related to the "scalability" of the model as shown by the ogn-products results ( the DNN state-of-the-art is around 90% here). So, I am curious whether this is the right use case for the SGHormerVQ model.

Thank you for your time and thorough comments.

(W1), (W3): "Motivation for spiking vector quantization is not clear enough...why consider spiking neurons? The process is similar to previous random feature propagation, what if we directly use it without the spiking function? Can the authors distinguish the contributions of this work more clearly?

R1: In the general responses, we restate our observations that converting neighborhood messages in multiple propagation steps into spiking vectors captures the graph structural information while replacing the original real-value representations space with a narrower spiking representation space. Inspired by these observations, we propose the spiking vector quantization to address issues present in previous VQ-based GNNs:

• The lack of graph inductive bias. For the vanilla vector quantization block derived from VQ-VAE [1], its predefined codebook doesn't involve any strong inductive biases. The emerging study [2] finds that structrue-aware tokenizers can be learned by adding the extra edge reconstruction loss. Benefiting from the characteristics of spiking neurons, SVQ actively captures the dynamic during the message propagation to generate a codebook with graph inductive bias.

• Codebook collapse. By learning a spiking function rather than pre-defining a codebook, SVQ can be seen as finding the optimal quantizer to map nodes into corresponding codewords. It effectively addresses the problem of codebook collapse.

In Appendix A, we perform SVQ on Cora, Citeseer and Pubmed datasets, and provide the visualization results of rate coded embeddings. As depicted in Figure 5 of Appendix A, directly performing random propagation without the spiking function on graph datasets generates high-precision representations for each node. Feeding the messages generated during multiple propagation steps into spiking neurons will generate the low-precision tokens of nodes, that nodes in the same class tend to be represented by the same spiking vector.

(W1): "there is no discussion on the issue of deployment on neuromorphic hardware...It is also not proper to mark the model as spike-based since some major parts are not spike-based."

R2: For SGHormerVQ, the spiking vector quantization can be deployed on the neuromorphic hardware. Notably, the SVQ can be decoupled with other modules, and the codewords corresponding to nodes can be pre-calculated before the inference phase rather than looking up entries in the codebook. It enables SGHormerVQ to generate and store the codebook with low energy consumption. Essentially, our model revisits the role of spiking neurons in the perspective of vector quantization and exhibits how spiking vector quantization results can be incorporated into GNNs. Therefore, we still refer to SGHormerVQ as the spike-driven architecture.

(W2): The claim "518x faster inference speed compared to other GTs" is not proper since SGFormer has a similar inference time. It is also unclear why inference speed is largely improved.

R3: Thank you for your helpful advice. Compared to GOAT, SGHormerVQ with better performance infers faster than this VQ-based GT by 518x on the Physics dataset. We apologize for the confusion and have revised this unclear description. In addition, we provided a comprehensive energy efficiency analysis to evaluate memory usage, inference latency and theoretical consumption of GTs in Appendix B.

The pre-calculated codebook in SVQ and the linear-time Transformer guided by codebook bring the significant improvement in inference speed.

• In SVQ, the codewords corresponding to nodes can be pre-calculated by feeding features into spiking neurons before the inference phase, rather than looking up entries in the codebook during the inference process.

• As described in the line 299, The complexity of CGSA is $O(NBd)$. The size of the codebook, $B$, also affects the inference speed of SGHormerVQ. As depicted in Figure 4, the codebook generated by spiking neurons is much smaller in size compared to predefined codebooks. In some cases, the codebook size $B$ is even smaller than the embedding dimension $d$, which enables the inference latency of our model to be slightly lower than SGFormer $O(Ndd)$.

(W3): Can the authors distinguish the contributions of this work more clearly?

R4: We would like to explain the core ideas of this work in detail. For SGHoremrVQ, one of the main contributions lies in proposing an efficient spike-driven vector quantization (SVQ) method for graph data. Its core idea involves feeding neighborhood messages from multiple propagation steps into spiking neurons to generate codewords .

It is worth noting that the message propagation algorithms in SVQ are model-agnostic. Replacing random feature propagation with sophisticated message-passing models (GCN, SAGE and GAT) is feasible. We believe that combining advanced message-passing mechanisms with spiking neurons can better capture the structural information of graphs and generate codebooks with graph inductive bias. To balance the efficiency and predictive performance of SGHormerVQ, this work adopts a relatively simple implementation.

The architecture design of a Codebook-Guided Transformer is not the primary goal of this work. Instead, compared to previous VQ-based GTs, our model actively injects the graph inductive bias to generate codebook and indirectly incorporates graph structural information into the Transformer.

(W4): There is no theoretical analysis for the proposed method.

R5: As mentioned in the general response, SGHormerVQ is largely inspired by proposed observations. In SGHormerVQ, we are more focused on revisiting spiking graph neural networks in the perspective of vector quantization rather than solving theoretical problems regarding SGNNs. The theoretical insight mainly comes from previous studies [1][2][3].

(Q1) For SNNs, spiking is only one of the properties. Very early works focused on analyzing their stronger computational power induced by the temporal dimension of spiking time, and recent works have introduced this thought to graph AI tasks. This paper only considers rate coding that loses a lot of information of spike trains. Is it possible to introduce these spiking time information to better enhance the codebook coding?

R6: Thank you for the inspired suggestion. It drives us to explore the spiking vector quantization based on temporal coding. The visualization results of decoding outputs from RGSNN and SVQ are demonstrated in Figure 6 of Appendix C. Since GRSNN is designed for link prediction tasks and the properties of edges are plain on node classification datasets, we also assign random features to nodes except for the embedding of relations. The outputs combined edge embeddings containing the temporal delay information with node embeddings will be fed into a mean aggregator to generate the predictive results. The results show that GRSNN integrating temporal delays in output spikes does provide more expressive spiking vectors. However, reconstructing more expressive node embeddings via spiking time as additional dimension conflicts with SVQ, which aims at mapping different nodes into the same low-precision spiking vectors.

[1] Li J, et al. A graph is worth 1-bit spikes: When graph contrastive learning meets spiking neural networks. ICLR, 2024.

[2] Eshraghian J K, et al. Training spiking neural networks using lessons from deep learning. arXiv, 2021.

[3] Gerstner W, et al. Neuronal dynamics: From single neurons to networks and models of cognition. 2014

We appreciate the reviewer for detailed comments and suggestive feedback. We want to clarify some misunderstandings that caused some of your concerns.

(W1): "However, this is actually just a spike train, which is the activation value of a spiking neuron, and it is difficult to apply a quantization technique that uses a codebook because it changes dynamically."

R1: First of all, we present the definition of rate coded outputs. $\hat{S}[t]=\Theta(V[t-1], I[t])\in{\\{0,1\\}}^D$ is a time-varying vector that represents the spikes emitted from each output neuron across time, where $\Theta(\cdot)$ is the Heaviside function. Let $S=\sum_{t=0}^T{\hat{S}[t]}\in\mathbb{Z}^D$ be the spike count from each output neuron, which can be obtained by summing $\hat{S}[t]$ over $T$ time steps. For $D=1, T=3$ and the input sequence is $I=[0.6, 0.5, 1.0]$, the rate coded output will be $S=2$. As described in Eq.(10)-(12), SVQ collects the messages $M[t]$ during the multiple propagation steps as input currents $I[t]$ across time, and converts the input sequences to integers. Considering spiking neurons as quantizers, we can utilize low-precision rate coded outputs (we refer to them as "low-precision spiking vectors" in the paper) to represent different nodes. This is also one of the contributions of our paper. We add additional visualization results in Appendix A to detail the role of spiking neurons during the feature propagation process.

(W1) " the representation space of the codebook should be narrower than the representation space of the activation. However, in this study, since these two are used identically"

R2: In SGHormerVQ, the representation space of the implicit codebook is much narrower than the representation space of the activation. For $D=3, T=2$, and node features can be seen as the input at $t=0$, $I[0]\in\mathbb{R}^3$, SVQ generates the input sequence $I\in\mathbb{R}^{2\times3}$ and mapped it into a implicit codebook $\tilde{C}=\\{(0,0,0),(0,0,1),(0,0,2),...,(2,2,2)\\}$, where $|\tilde{C}|=(T+1)^D=27$. SVQ quantizes node representations to a finite set of codewords. More details have been explained in lines 255-257.

(W1) "vector quantization using an implicit codebook simply processes information with spike activation. It is unreasonable to view the improvement as being due to a codebook vector quantization. The main reason for the improvement is believed to be due to the characteristics of spiking neural networks that process information with binary spikes. The authors need to clarify this."

R3: We believe that the improvement in predictive performances of SGHormerVQ is brought from the fusion of SVQ and CGSA modules. We will clarify the role of each module in SGHormerVQ:

• SVQ: Figure 1 shows that SVQ mapps different node representations into same low-precision spiking vectors by combining the random feature propagation with spiking neurons. Those nodes represented by same spiking vector can be regarded as having similar neighborhood structures. These lower-space vectors require less storage space, which brings an improvement in efficiency.
• CGSA: By replacing the original key matrix with spiking vectors, the vanilla attention from nodes to nodes can be transformed into a linear-time attention from nodes to grouped node sets. The CGSA module effectively captures long-range structural information and generate more expressive node embeddings.

(Q1): Why is the title SGHormerVQ?

R4: For differentiating the existing GT method [1] and highlighting the spiking vector quantization module, we named our method SGHormerVQ (Spiking GrapH transfORMER via a Vector Quantization variant).

(Q2), (Q3): What algorithm was used for learning? What surrogate function was used?

R5: The backpropagation algorithm is used to update the model parameters. Besides, as mentioned in the line 186, we adopt surrogate gradients during error backpropagation to address the issue of zeros gradients caused by non-differentiable functions in Eq.(12). In the implementation, we adopt the Sigmoid function as the surrogate function.

(Q4): What is the effect of the norm function in Equation 11?

R6: As mentioned in the line 269, the $Norm(\cdot)$ in Equation 11 is responsible for normalizing output messages to the range of threshold membrane potential $V_{th}$.

(Q5), (Q7): What is $R$ in g{V,e,X} → g{V,e,R} in Figure 2? Line 305: "we construct random features" → What does it mean?

R7: In line 254, we detail that $R$ is denoted as the random features, which is a $D$-dimension matrix sampled from the uniform distribution. And random features are widely used to improve the expressiveness of MPNNs [2][3].

(Q6): What does U in Equation 13 represent? It is a one-hot matrix, so what information does it contain?

R8: $U$ can be seen as the indices for where the codewords in the rate coded embeddings $S$ ended up in the unique matrix $C$. And $C$ can be seen as removing duplicate vectors from $S$. Specifically, as describe in Eq.(13), the rate coded embeddings $S$ can be decomposed into the codebook $C\in\mathbb{Z}^{B\times D}$ and the one-hot matrix $U\in{\\{0,1\\}}^{N\times B}$, where $B$ is the size of reconstruted codebok, $B\ll|\tilde{C}|$ and $D$ is the dimension of random features.

(Q8): Is SVQ the only part that uses spiking neurons? It seems that information is not processed by spikes in other parts. Then, how can this model work on neuromorphic hardware?

R9: In SGHormerVQ, the spiking vector quantization module driven by spiking neurons can be deployed on specific neuromorphic hardware. We notice that different from some existing spiking-driven Transformer methods in computer vision [4], directly converting the output embeddings from Transformer blocks or GNN blocks into the spiking format will seriously impair the predictive performance of models. In SGHormerVQ, the reconstructed codebook $C$ is derived from the rate coded embeddings, which means we can feed random features into neuromorphic hardware to generate the codewords corresponding to nodes with lower energy consumption. Additionally, we provided a comprehensive energy efficiency analysis in Appendix B.

[1] Wu Q, et al. SGFormer: Single-Layer Graph Transformers with Approximation-Free Linear Complexity. NIPS, 2024.

[2] Yao M, et al. Spike-driven transformer. NIPS, 2024.

[3] Puny O, et al. Global attention improves graph networks generalization. arXiv, 2020.

[4] Dasoulas G, et al. Coloring graph neural networks for node disambiguation. arXiv, 2019.

Dear Area Chairs and Reviewers,

We sincerely thank all the reviewers for their valuable feedback. In response to the remaining concerns, we have revised the paper to better highlight its significance. The new changes are marked in red, while the previous changes are marked in orange. The specific modifications in this final revision are as follows:

1. We provide extra visualization results and corresponding explanations in Appendix A. Besides, we update these visualization results to better distinguish floating point node embeddings and integer rate-coded vectors.

(NQbN) "Personally I found your discussion in Appendix A to be very interesting, but too short - would it be possible for the authors to add more details of the comparisons and how they are interpreting the visualizations etc."

2. We compare the high-precision embeddings during the message propagation and the corresponding rate-coded vectors in Figure 5 of Appendix A to highlight the necessity of spiking neurons.

(BudY) "it is unclear why spiking is necessary or what is the advantage of the claimed "low-precision tokens"

3. We refer to the spike count outputs obtained by summing spiking embeddings over $T$ propagation steps as rate-coded vectors, to explicitly differentiate them from spiking activations which consist of 0 and 1 elements.

(oc9C) "The implicit codebook is simply the same as the activations of spiking neurons. It is more reasonable to consider this as binarization rather than a codebook for quantization."