Spiking Transformer with Experts Mixture

Thank you for your comments and suggestions for improvement.

Weakness 1: Several Unclear Notations.

AW1: Sorry for the confusion caused by unclear notations. To clarify: (1) $N$ denotes the length of image patches. (2) Thanks for your advice. We will change the abbreviation representing the depth-wise convolutional layer from "DWL" to "DWC". (3) The input current matrix (feature) $I[t]$ has a dimension of $P\times Q$, where p and q represent the $p$-th row and $q$-th column, respectively. We will modify them in the manuscript.

Weakness 2: The Section 4.3 is more like a sensitivity experiment of hyper-parameters rather than an ablation study.

AW2: We appreciate your suggestion and will revise the titles of the ablation study and the hyperparameter sensitivity experiment in the manuscript accordingly.

Question 1: Can you discuss the feasibility of adapting your proposed EMSA and EMSP modules to neuromorphic hardware?

AQ1: SEMM can theoretically be deployed on chips that are synchronous within layers and asynchronous between layers, such as TrueNorth [R1]. The spike-driven operation of SEMM is similar to the AND and ADD residuals in SEW ResNet [R2]. Multiple branches and residuals are supported on certain neuromorphic hardware, such as Speck V2 [R3]. However, deploying SEMM on a fully asynchronous neuromorphic chip is challenging.

Question 2: How does SEMM handle the trade-off between sparsity and accuracy?

AQ2: In SEMM, we do not control sparsity using hyperparameters; instead, the model dynamically adjusts based on the input. Therefore, there is no trade-off between sparsity and performance. Higher sparsity does not necessarily imply lower accuracy. SEMM utilizes dynamic spiking sparsity to select feature representations within each expert, thereby enhancing accuracy.

Limitation 1: While the authors indicate that the limitations are addressed in Section 5 (refer to Line 533 in the Checklist), it appears that this section does not currently include a discussion on the limitations. Can you include a discussion on the limitations of the work?

AL1: We apologize for the confusion caused by our incomplete statement. In Sec. 5, we state that future work will focus on SEMM application on larger SNN models. In line with your suggestion, we further discuss the limitations of this work:

SEMM has been validated on image datasets, demonstrating its effectiveness. However, no further verification has been conducted on other tasks and diverse datasets. Despite validating SEMM's effectiveness in time series forecasting tasks as suggested by reviewer RYQd, demonstrating its generalizability, there remain numerous tasks yet to be validated. Additionally, the effectiveness of SEMM on larger SNN models has yet to be validated. Future work will address these two limitations.

We will add this discussion to the appendix of the manuscript.

[R1] Akopyan, Filipp, et al. "Truenorth: Design and tool flow of a 65 mw 1 million neuron programmable neurosynaptic chip." IEEE transactions on computer-aided design of integrated circuits and systems 34.10 (2015): 1537-1557.

[R2] Fang, Wei, et al. "Deep residual learning in spiking neural networks." Advances in Neural Information Processing Systems 34 (2021): 21056-21069.

[R3] Ole, Richter, et al. "Speck: A Smart event-based Vision Sensor with a low latency 327K Neuron Convolutional Neuronal Network Processing Pipeline." IEEE International Symposium On Asynchronous Circuits and Systems (2023).

Dear Reviewer,

Thank you again for your valuable time and feedback.

Best regards

Authors

We appreciate your detailed comments. We would like to address your concerns below.

Weakness 1: The author needs to provide a more detailed explanation of the spike-driven advantage, especially on asynchronous hardware.

AW1: As you mentioned, deploying SEMM on a fully asynchronous neuromorphic chip is challenging. However, SEMM can theoretically be deployed on chips that are synchronous within layers and asynchronous between layers, such as TrueNorth [R1]. Specifically, multiple spiking router matrices, i.e. $\{ r_1,r_2,...,r_m\}$, can be implemented by one block (Linear-BatchNorm-SpikingNeuron) and thus can be mapped to the one core of the neuromorphic chip for computation. Outputs from the same core are considered synchronous. The spiking Hadamard product and addition that occur in SEMM are analogous to the AND and ADD operations in SEW ResNet [R2]. These element-wise spiking operations can be adapted on the neuromorphic chip that supports multiple branches and residuals, such as Speck V2 [R3].

Weakness 2: Prior to Eqn. 14, the authors omitted the equation for partitioning the input X (how to get the $A_m$?).

AW2: Sorry for the confusion caused by ommitation. In Eqn. 14, Each $\bf A_m$ is obtained as follows,

$${\bf A\_m} = \text{SSA}(\bf Q\_m,K,V),$$

where $\bf Q_m$ is individual to each expert, and $\bf K, V$ are shared among experts. We will revise it in the manuscript.

Weakness 3, Question 1: In Eqn. 17, $BN(SEMM(E, R, F(·)))W_o$ undoubtedly involves floating-point matrix operations; is there a writing error present here? The SEMM(E, R, F(·)) obtained from Eqn. 16 is described as containing only 0 or 1. But it should be a floating-point matrix.

AW3: We apologize for our incorrect expression. The $\text{F}(·)$ in Eqn. 16 involves spiking Hadamard products and element-wise addition. Since both $\bf E$ and $\mathcal{R}$ are in spiking form, the output of ${\rm{SEMM}}({\bf E}, \mathcal{R}, {\text F}(\cdot))$ is an integer spiking form rather than a binary spiking form. $\text{BN}({\rm{SEMM}}({\bf E}, \mathcal{R}, {\text F}(\cdot)))\bf W_o$ in Eqn. 17 can be decomposed into addition operations and multiplication is avoided, which is shown as ${\rm{SEMM}}({\bf E}, \mathcal{R}, {\text F}(\cdot)) = \sum_{i=1}^{m} {\bf r}_i * {{\bf A}_i} = \bf s_1 + s_2 + \cdots + s_m$ ${\rm{SEMM}}({\bf E}, \mathcal{R}, {\text F}(\cdot))\bf W_o = ({\bf s_1 + s_2 + \cdots + s_m})\bf W_o = \bf s_1W_o + \bf s_2W_o + \cdots + \bf{s_m}W_o$ where $\bf s_m$ is binary spiking form. We omit $\text{BN}$ because its parameters can be integrated with the linear layer. We will revise it in the manuscript.

Question 2: Line 217 What means "The ASR of EMSA is around 0.5, which is comparable to the regular TopK setting of ANN-MoE"? What's the relationship between the ASR in EMSA and the TopK rate in ANN-MoE?

AQ2: In ANN-MoE [R4, R5], TopK typically selects 2 experts, which can be considered as 50% sparsity for a total of four experts in our settings. This is similar to the average spiking rate (sparsity) observed in EMSA.

Question 3: What is the purpose of Fig. 6? In my view, different inputs leading to different activations are natural.

AQ3: The purpose of Fig. 6 is to further demonstrate the dynamic condition computing characteristics of routers in SEMM. It illustrates the average spiking rate across spatial-temporal locations of routers across different classes of images (with 50 images per class), offering a more representative visualization compared to the single image in Fig. 5. Specifically, the spiking pattern of a router varies with time steps and spatial positions. Across different categories, the active routers differ; for example, in (a), routers 1 and 2 (top left and top right) are active, while in (b), routers 1 and 4 (top left and bottom right) exhibit high activation.

[R1] Akopyan, Filipp, et al. "Truenorth: Design and tool flow of a 65 mw 1 million neuron programmable neurosynaptic chip." IEEE transactions on computer-aided design of integrated circuits and systems 34.10 (2015): 1537-1557.

[R2] Fang, Wei, et al. "Deep residual learning in spiking neural networks." Advances in Neural Information Processing Systems 34 (2021): 21056-21069.

[R3] Ole, Richter, et al. "Speck: A Smart event-based Vision Sensor with a low latency 327K Neuron Convolutional Neuronal Network Processing Pipeline." IEEE International Symposium On Asynchronous Circuits and Systems (2023).

[R4] Noam Shazeer, et al. "Outrageously large neural networks: The sparsely-gated mixtureof-experts layer." International Conference on Learning Representations (2017).

[R5] Yanqi Zhou, et al. "Mixture-of-experts with expert choice routing." Advances in Neural Information Processing Systems (2022).

Dear Reviewer,

Thank you for your recognition of this work.

Best regards

Authors

Thank you for your detailed comments and suggestions for improvement. We would like to address your concerns and answer your questions below.

Weakness 1: The application of SEMM on the most state-of-the-art Spiking Transformers such as Spike-driven Transformer V2.

AW1: Following your suggestion, we report the Spike-driven Transformer V2 (SDT V2) with SEMM on CIFAR100 and ImageNet in Tab. R1-1.

Table R1-1

After configuring SEMM, SDT V2 demonstrates a consistent performance improvement, but the degree of improvement is not as large as that of SDT V1. This could be attributed to SDT V2 being a hybrid spiking CNN-Transformer framework, where the first two stages are based on spiking convolutions, with the third and fourth stages utilizing spiking transformers. Therefore, compared to SDT V1, the proportion of EMSA and EMSP blocks in SDT V2 is lower, resulting in a smaller gain. We will add these results to the manuscript appendix.

Furthermore, we argue that the SEMM differs from the framework design of Spiking Transformers (Spikformer, Spike-driven Transformer V1/2, and Spikingformer, et al.). SEMM is a universal plug-and-play paradigm that can stably enhance performance. It provides a new direction for improving the Spiking Transformer.

Weakness 2: Provide evidence demonstrating the effectiveness of using DWConv after MLP.fc1.

AW2: In EMSP, the role of DWConv is to independently extract features at the channel-level experts using a 3x3 receptive field, thereby enhancing representational capacity. The ablation study about DWConv in Tab. R1-2 demonstrates its effectiveness:

Table R1-2

Thanks for your comments, we will include the ablation experiments in the manuscript appendix.

Weakness 3: Is the ANN-MoE paradigm derived from prior ANN-MoE studies?

AW3: Thank you for your kind reminder. ANN-MoE represents a paradigm derived from previous works on MoE in ANNs [R1, R2], integrating multiple experts through routing and hard sparsification. The referenced ANN MoE works in the manuscript adopt this paradigm, for example, manuscript line 32 "Mixture-of-Experts (MoE) [20; 21] is known for allowing each expert to learn specific tasks or features, showing better performance" and line 89 "The Mixture-of-Experts (MoE) [28; 29] combines the predictions of multiple specialized experts". For clarity, we will append citations to these works following the term 'ANN-MoE' in the manuscript.

Weakness 4: The description for Figure 3 is not clear enough.

AW4: Fig. 3 depicts the parameter number (X-axis) and performance (Y-axis) of various MoE strategies under multiple Spiking Transformer baselines. Each line consists of four points from left to right, representing:

• 1. Baseline
• 2. EMSP
• 3. ANN-MoE paradigm with float-point router matrix (AM-F)
• 4. ANN-MoE paradigm with spiking router matrix (AM-S)

For clarity, we convert Fig. 3 into the following Tab. R1-3. It can be observed that our EMSP achieves better performance with fewer parameters, whereas the application of the ANN-MoE paradigm yields suboptimal results. We will revise Fig. 3 in the manuscript to enhance its readability.

Table R1-3

Weakness 5: In Figure 4, do the four graphs in each sub-figure relate to four images from the same class?

AW5: We speculate that you are referring to Fig. 5, not Fig. 4. The background image is the same for each subplot. We will add this explanation to the manuscript.

[R1] Noam Shazeer, et al. "Outrageously large neural networks: The sparsely-gated mixtureof-experts layer." International Conference on Learning Representations (2017).

[R2] Yanqi Zhou, et al. "Mixture-of-experts with expert choice routing." Advances in Neural Information Processing Systems (2022).

Dear Reviewer,

Based on your suggestion, after overcoming significant challenges in resource allocation, we provide a performance report of using SEMM on the largest Meta-SDT-V2 model:

Table R1-4

Here, DT stands for Direct Training, while KD denotes Knowledge Distillation Assisted Training. SDT-V2-54M with SEMM DT shows an improvement of 2.10% over the direct training of the original model, and 1.80% higher than the knowledge distillation training of the original model.

SEMM can still achieve stable performance improvements on the largest Meta-SDT-V2 model while maintaining a similar number of model parameters, demonstrating its generalizability and effectiveness.

The logs, accuracy curves, specific code, and weights have been posted on an anonymous external link: https://huggingface.co/SDTV2SEMM/Rebuttal-SDTV2-SEMM-55M

Best regards

Authors

Thanks for your valuable comments. We will explain and discuss your concerns.

Weakness 1, Question 1, Limitation 1: More experiments across diverse datasets to generalize the findings.

A1: These datasets in experiments possess distinct characteristics and scales, which verify the effectiveness and generalization capability of SEMM. Specifically, the neuromorphic dataset exhibits temporal characteristics and sparse event features, validating SEMM's capability to handle sparse event data. The ImageNet-1K, with its 1,000 classes, verifies SEMM's ability to process complex large-scale data. In terms of data scale, we apologize for not being able to provide results on larger datasets due to the rebuttal time and resource constraints. In terms of data diversity, we add the validity and generalization experiments of SEMM on time series forecasting, which is a challenging task and aims to predict future values based on historical observations arranged chronologically. Addressing this task often involves modeling the temporal dynamics, resonating profoundly with the nature of neural coding and SNN. As shown in Tab. R2-1, SEMM demonstrates superior predictive capabilities over the baseline on four datasets, further illustrating its versatility and robustness.

Table R2-1

Here, the horizon represents the prediction length. RSE stands for Root Relative Squared Error, where a smaller value indicates a lower error. $\text R^2$ represents the coefficient of determination, with higher values indicating higher prediction accuracy.

Weakness 2: The lack of theoretical guarantees or analysis regarding the convergence and stability of SEMM leaves room for skepticism about its robustness.

AW2: We regret that we are unable to provide a theoretical analysis of the convergence and robustness of SEMM. To the best of our knowledge, there are currently no works theoretically analyzing the convergence of deep SNNs. However, to address your concern, we validate SEMM's convergence and robustness through experiments.

Convergence: From an experimental perspective, SEMM does not affect the final convergence results of the spiking transformer. In Fig. R2-1 and R2-2 of our submitted PDF rebuttal document, we compare the training loss and test accuracy curves between the Spikformer baseline and SEMM. The introduction of SEMM did not affect convergence.

Robustness: From an experimental perspective, SEMM does not compromise the robustness of the model. To show this, we add experiments about input noise on CIFAR100. We add a Gaussian noise with mean 0 and variance $\sigma^2$ on inputs $X$. The results are shown in Tab. R2-2. It can be found that at each noise level, introducing SEMM is still better than the baseline.

Table R2-2

Weakness 3: A more detailed discussion on computational efficiency and scalability with respect to dataset size and complexity would be beneficial.

AW3: Thanks for your comments. Following your suggestion, we show that SEMM exhibits higher computational efficiency compared to ANN-MoE paradigm on three diverse datasets, i.e., CIFAR100 (static images), CIFAR10-DVS (neuromorphic images), and Solar (time series data). As shown in Tab.R2-3 (following), the number of parameters (Param) and computational operations (OPs) of SEMM on the three datasets are both lower than that of ANN-MoE, while also achieving the best performance. This demonstrates its computational efficiency and scalability for different datasets.

Table R2-3

Question 2: How does the proposed SEMM adapt to varying levels of sparsity in the input data, and what is the impact on computational efficiency?

AQ2: Regarding question 2, could you further explain what you mean by data sparsity?

Limitation 2: Implementation Details for Reproducibility.

AL2: You can refer to the Appendix. C for the hyperparameters and find the source code of SEMM in the supplementary materials.

[R1] Li, Yaguang, et al. "Diffusion convolutional recurrent neural network: Data-driven traffic forecasting." arXiv preprint arXiv:1707.01926 (2017).

[R2] Lai, Guokun, et al. "Modeling long-and short-term temporal patterns with deep neural networks." The 41st international ACM SIGIR conference on research & development in information retrieval (2018).

Dear Reviewer,

We hope this message finds you well. We have provided thorough responses to you and sincerely hope you can look through them and update the scores if your concerns have been resolved. We are also open to further discussion if the concerns have not been fully addressed. Please feel free to let us know if you still have any questions.

Best regards

Authors

Dear Reviewer, Thank you for your time and effort in reviewing our work. We will explain and discuss your concerns.

Concern 1: Explain the statement "SEMM has a variable sparsity when dealing with different data".

AW1: Sorry for the confusion caused by the unclear statement. We intend to convey that SEMM exhibits variable sparsity when dealing with different data within the same dataset. As shown in Fig. 5,6, and Tab. 2, for different images, each expert's corresponding router selects a different number and position of tokens, while the overall low firing rate indicates the sparsity.

Concern 2: Show the final contributions or advantages of "sparse conditional computation".

AW2: Similar to ANN-MoE mechanisms, the sparsity conditional computation characteristic of SEMM serves as the source of the performance improvement and computational efficiency achievement. Our experiments show that SEMM can bring stable performance improvements to SNN Transformers.

In terms of computational efficiency, as shown in Tab.R2-3, SEMM results in a substantial reduction in both parameters and operations (OPs), approaching the level of the baseline when compared to the introduction of ANN-MoE mechanisms.

Concern 3: Evaluation on larger datasets.

AW3: We apologize for not being able to provide results on larger datasets due to the resource constraints. We will add discussions on this issue in the Limitation.

Concern 4: Only the LIF neurons are not so strong to support the novelty of the proposed model.

AW4: Unlike ANN-MoE, the spike-form routers and experts in SEMM, as well as their integration process, are all in line with the spike-driven characteristics of SNNs. At the same time, SEMM also achieves a similar effect of sparse conditional computation as ANN-MoE, thereby stably enhancing performance on SNN Transformers. In addition, the lightweight architectural design is also an advantage over ANN-MoE. As shown in various experiments, the number of parameters and the number of operations after introducing SEMM are close to the baselines, far less than that of ANN-MoE introduction.

Best regards

Authors