SpikeGPT: Generative Pre-trained Language Model with Spiking Neural Networks

Authors can add a discussion about SpikeGPT's performance on specific hardware and its adaptability to different hardware platforms. The novelty is limited, I just did not see some crucial difference between general GPT and the proposed model about models and training methods. Another more important thing is that I did not see a detailed power consumption analysis on neuromorphic hardware, this work is different from other SNN based studies, large scale model must applied to true neuromorphic hardware which could analyze the true power consumption. As illustrated in table 2, the parameters of the proposed model are more than other models, I did not see the advantages of the spiking version of GPT. Furthermore, it would be beneficial to include additional datasets to demonstrate the model's effectiveness. Ablation experiments can be extended to include other evaluation metrics mentioned in the tables, such as accuracy and complexity, to provide a more comprehensive assessment of the model's efficacy.

1. The main innovation points are limited, this paper wanted to grasp this concept, but it didn't delve deeply into it. The network structure is more likely a hybrid one with transformer, not a true spike one.
2. The training methods for this model are not described in a clear way, so I did not see more novelty in this work.
3. The blocks in SRFFN seem too simple, I mean the authors should consider more about other gates such as forgetting gates.
4. During the training and inference phase, I did not see the true contribution of this model in NLG and NLU.
5. In table 1, the energy consumption used by authors is wrong, this model is not completely a spike based one (just inputs are spikes), in hence, used FLOPS and methods (Rathi and Roy 2021) are not fair.
6. Table 2 reports the complexity and parameters of some models, just from parameters, I can't see where the advantage lies (transformer vs spikegpt). Why don't I use another spike transformer model?
7. The authors make the comparison between spikegpt and gpt-2, gpt 2 is not the current one, so it is not an efficient comparison.

1. SpikeGPT introduces some float-point operations including float-point multiplication, division, and exponentiation. This makes SpikeGPT different from traditional spiking neural networks. Although these operations are not dominant in terms of computational overhead, this part of the operation will not be sparse and event-driven, which puts a higher demand on possible application scenarios that must support this hybrid computational paradigm.

2. The normalization used in SpikeGPT is unclear. As shown in the left subplot of Fig. 1, Add&Norm is used in SpikeGPT, but it is not specified in the text whether it is a layernorm or a batchnorm (or other normalization). Batchnorm is generally used for SNNs because it can be merged into linear or convolutional layers, whereas layernorms, which are widely used in NLP, cannot be merged, which results in extra floating-point multiplications and is not applicable to typical SNNs. Based on the code given by the authors, I would guess that the authors use layernorm, but the authors don't provide any analysis of the computational overhead of normalization and the effect of layer normalization on spike inputs in the main text. In addition, Eq. 3 and Eq. 4 do not mention the normalization.

3. Estimates of energy efficiency in the main text are rough. First, the energy consumption estimates in Table 1 assume that all spiking neurons have the same firing rate of 0.15. The authors do not explain how the 0.15 is derived here, but it is safe to assume that this is only a rough estimate, since the firing rate of neurons in different layers and even different channels of the network should be different. Second, the energy consumption estimates in Table 1 do not seem to take into account the differences in energy consumption for different kinds of floating-point operations. In spiking RWKV, the element-wise float-point, division, and exponentiation (including sigmoid) should have different energy consumption with the MAC.

4. Missing further analysis of $\rm {ReLU}^2$ activation. In page 7, line 5, section 3.7, the authors state that "While $\rm {ReLU}^2$ activations are not binary, they induce dynamical sparsity." However, the authors do not give an estimate of this sparsity. In addition, the energy estimates in Table 1 do not take into account the energy consumption of $\rm {ReLU}^2$, nor do they use the sparsity of $\rm {ReLU}^2$ with $\rm E_{MAC}$ to estimate the energy consumption of the linear transformation $\bf M_s$.

5. The contribution on improving RWKV is unclear. The authors only review vanilla self-attention in the main text, but instead of reviewing the RWKV structure they put this part in the appendix. In addition, the authors do not clearly indicate which parts are the authors' main contributions and which are RWKV contributions in the main text. Therefore I need to carefully compare the content about RWKV in the appendix to determine the author's main contribution. As far as I understand from the main text, the author's contribution to the improvement of the RWKV structure is as follows:

   • They add a spiking neuron layer before the linear layer (or equally, add a spiking neuron layer after block output) to reduce the computational overhead of the linear layer.

   • They modify the mechanism of the positional weight decay in RWKV.

I am not sure what is the contribution of this paper besides the fact that you have spikified a GPT model.

Further, I dont agree with author's claims on lightweight. All their complexity evaluations are analystical. Can the authors give quantitaive speedup numbers say by running on an actual GPU/TPU hardware whether their spikeGPT gives faster results than many other lightweight GPTs that exist today, such as miniGPT?