Time to Spike? Understanding the Representational Power of Spiking Neural Networks in Discrete Time

We thank Reviewer MuQ3 for the feedback. Below we individually address the (not discussed) concerns:

1. Abstract/title

We agree that the focus on static data should be highlighted in the abstract (instead of only in the introduction as of now). We will acknowledge this in the abstract of the revised version. Concerning the title, with 'time' we actually referred to the temporal aspects in the neuronal dynamics itself, i.e. the importance of time in the propagation of information via binary spikes.

2. Mismatch between LIF models

As of today, reset-by-subtraction seems to be the more common choice for the reset mechanism (it is the standard setting in many neuromorphic software platforms such as snntorch or SpikingJelly) in comparison to reset-to-zero (or to a fixed value). Experimental evidence and heuristic reasons (often to avoid information loss) for this choice can be found e.g. in [1], [Eshraghian et al., 2023], [Rueckauer et al., 2017]. Also, we believe that extending our results to the other mechanism is feasible and possibly not too difficult to achieve.

[1]: RMP-SNN: Residual Membrane Potential Neuron for Enabling Deeper High-Accuracy and Low-Latency Spiking Neural Network, Han, Srinivasan, Roy, CVPR 2020

3. Proof sketch issue

We believe that the reviewer is referring to Thm 3.2 rather than Prop 3.1. In this case, the main points of the sketch are (1) continuous functions can be approximated by step functions, and (2) step functions can be realized by discrete-time LIF-SNNs. Such an argument is quite common in deep learning literature on expressivity. In the revised version, we will take into account the reviewer's high-level suggestion to improve the clarity of this proof sketch.

4. Significance of Prop 3.3

In Prop 3.3, the first statement is indeed a direct consequence of Thm 3.2, while the second statement requires a different construction. Basically, the result answers the question of whether the constructive approximation derived in Thm 3.2 is optimal by showing in the worst-case scenario that the number of required neurons is minimal up to constant factors. In the formulation of Prop 3.3, we repeated the result from Thm 3.2 in the first part to facilitate the comparison between upper and lower bounds. As the reviewer noted, this proposition is not applied anywhere else. It is intended as an extension of Theorem 3.2, showing the optimality of the approximation.

5. Possible misleading comparison with ReLU case

In our analysis, we do not distinguish between the leaky and non-leaky case. Thus, in both cases, the maximum number of regions could achieve the quadratic growth with respect to the latency. The mentioned remark compares the scaling of the maximum number of regions with respect to different hyperparameters, highlighting the role of depth and latency in the model complexity.

However, we think that just directly considering the maximum number of regions is not sufficient, especially when comparing the expressivity of SNNs to ReLU-ANNs. The main reason is while SNNs realize piecewise constant functions, ReLU-ANNs realize piecewise linear functions. Furthermore, it is worthwhile to clarify that other aspects besides the maximum number of regions are also relevant to the model expressivity. This is also the motivation for our ablation study in Section 5.2 and 5.3.

6. Experiments in Section 5.1

• Lack of test accuracy plots: As the reviewer also pointed out, we found these experimental results not relevant to the theoretical expressivity results and therefore excluded them from the paper. However we may add test accuracies to the revised paper for completeness.

• Choice of datasets: We conducted experiments on MNIST, but our observation was that MNIST is too simple to gain insights into expressivity as the training data is almost interpolated even with very small networks, which makes it difficult to reflect the improvement of the model expressivity along with the subsequent hidden layers' width. Therefore, we think that a more complex dataset like CIFAR10 is more appropriate. We also conducted similar experiments on SVHN, leading to results with the same trend as seen for CIFAR10 and will include these results in the revised version.

7. Experiments in Section 5.3

• Table 1: We agree that it would be useful to know how the number of regions scales. However, for now, our naive counting algorithm is based on grid search and may overlook several small regions when $T$ gets larger (given the observations in Section 5.2). In the revised version, we will incorporate the reviewer's suggestion by complementing the case $T=1$ as well as a few more different (small) values for $T$.

• Choice of toy example: Our goal is to show that the model may significantly reduce the number of regions during training. Choosing more complex data may be misaligned with this goal, while requiring larger networks and making region counting prohibitive.

We thank to Reviewer Wwin for the feedback.

We would like to start our response by clarifying—although this is stated multiple times in the paper (e.g. lines 21, 55, 100) as well as discussed explicitly in the related works section (e.g. lines 1466-1469)—that we are not proposing a new SNN or neuronal model. Rather, we are analyzing a well-known model and providing a rigorous mathematical framework for it. The reviewer may have been confused by our use of the term 'our model' on a few occasions. However, we used this term only after clarifying that the model is already widely used and was intended merely as a shorthand for the one analyzed in our paper-a stylistic choice in writing. We believe this is a common practice, but if it caused confusion, we are open to revising it in the final version of the paper.

Our motivation for formalizing the discussed SNN model in a rigorous framework is to ensure that our results are both mathematically sound, general, and understandable. It is important to note that several variants of the studied model exist (which are captured by our framework), differing in aspects such as which parameters are trainable. Comparisons with other neuron models have already been well established in numerous prior works e.g. [Stanojevic et al., 2024], [Fang et al., 2023], etc. In this regard, we consider the comparative experiments and experimental validation (in the sense suggested by the reviewer) to be beyond the scope of our paper.

On the other hand, this model is already widely used; for instance, it is essentially the one implemented in many neuromorphic software and hardware platforms such as [Eshraghian et al., 2023], [Fang et al., 2023], [Gonzalez et al., 2023]. Therefore, we find the concern about the feasibility of deployment on neuromorphic chips not valid.

Concerning the reviewer's comment on writing, the related work section is expanded in Appendix D, and we highlighted only the most relevant literature in the related work of the main part. Additionally, we could not resolve the formatting issues yet due to the fixed ICML template, however, this should not be an issue for a camera-ready version.

We thank Reviewer 9JUu for the feedback. Below we address each mentioned point:

1. Distinction from activation quantization studies

First, we confirm that our analysis in Section 3.2 indeed builds on existing results on Heaviside ANNs, as we point out at line 165 (but should be stated more clearly in a revised version). Moreover, we also discussed the relation to existing results for Heaviside ANNs from [Khalife et al., 2024]. Overall, the proof idea is adopted from the consideration of Heaviside ANNs in this paper, with necessary adjustments due to the considered function class and the specific type of result.

Second, concerning the ANN-SNN conversion and quantization perspective, we think that this is an interesting and valid viewpoint about SNNs. In fact, this perspective has been discussed in several previous works, e.g. [Eshraghian et al., 2023] or [1]. However, such discussions often do not offer a rigorous mathematical verification or are restricted to special cases. For instance, the work [1] shows the equivalence between ANNs with the quantization clip-floor activation and SNNs based on non-leaky IF neurons with rate coding. In this case, one could study the input partitioning of SNNs through the specific quantized ANNs. However, such an equivalence does not hold in more general cases, in particular in the case of LIF neuron model that we study in our paper. In other words, given a discrete-time SNN model, it is, in general, not clear how to get an exactly equivalent quantized ANN, through which one could better understand the original SNN. Nevertheless, we appreciate the reviewer's perspective about the link between quantized ANNs and SNNs and believe that describing this connection in a rigorous mathematical way would indeed provide meaningful insights and is an important future work.

2. Modeling of multi-step SNNs and its implications

First, the connection between Section 3 and 4 can be explained as follows. Given that discrete-time LIF-SNNs are universal, even in the 'trivial' case of $T=1$, one natural question is what the different hyperparameters, such as depth, width, and especially latency contribute specifically to the network expressivity (and beyond). This question is addressed in Section 4 by studying the model complexity measured by a reasonable metric, which is the richness of the input partitioning. In this sense, Section 4 can be seen as a natural continuation of Section 3, albeit now the latency by design plays a more crucial role compared to the previous analysis.

Second, we want to clarify the meaning of the last paragraph of Section 3. In general, this paragraph was intended to introduce the similarity (linear partitioning) between multi-step SNNs and Heaviside-ANNs as motivation for the upcoming Section 4. With "varying" or "flexible" biases, we indeed mean that the biases are input- and time-dependent, as pointed out by the reviewer. This connection is rigorously established Equation 13 (in the appendix) that we referred to in the paragraph. Moreover, we would like to emphasize that we do not assume any simplification of the temporal dependencies. In fact, our analysis accounts for such temporal dependencies and involves computing the exact bias in each time step in relation to inputs and weights. Dealing with the mentioned temporal dependencies is the main point in Lemma 4.2 (or repeated as Lemma B.13 in the appendix) and poses the main challenge in analyzing the actual number of linear regions.

Finally, we will try to clarify these issues in a revised version of the paper to avoid such confusion by modifying the mentioned paragraph.

3. Assumptions on temporal constraints and their impact

As stressed in the previous point, we do not assume any simplification on temporal constraints, in particular, we do not assume that each time step involves an independent network. In fact, since we took into account the temporal dynamics, the firing pattern of each neuron is not arbitrary (which would lead to $2^T$ different binary patterns from $\set{0,1}^T$ ), but is restricted to only $T^2$ possible binary codes. This shows that the temporal dynamic has a noticeable limiting effect on the number of regions created by the SNN, as discussed by the reviewer.

As a side note, we agree with the reviewer that the temporal constraints are helpful to understand the shift behavior observed in Section 5.2. In fact, we were able to prove (after the submission) that in the non-leaky case, the shift converges to the input current. In the leaky case, it seems that the shift still 'tries' to approximate the input current, but the approximation cannot be accurate. This leads to a periodic output spike pattern and thus the observed periodic shift behavior.

[1]: Optimal ANN-SNN Conversion for High-accuracy and Ultra-low-latency Spiking Neural Networks, T. Bu, W. Fang, J. Ding, P. Dai, Z. Yu and T. Huang, ICLR 2022.

We thank Reviewer V5so for the feedback. Below, we address the concerns individually:

1. Training and architectural design

These features of ANNs may benefit from expressivity insights [1,2, Appendix D.1], while today's SNN architectures and training strategies are typically based on those of ANNs. Given that SNNs function differently, one should reconsider the common practices of architectures and training (e.g., increasing depth and latency, applying convolution, batch normalization, pooling, or applying more complex neuron dynamics). Our analysis questions the possible benefits of certain architectural design techniques and relates them to the increase in computation and complexity expense in training and inference. Suggestions for improving the spatial and temporal components of SNN architectures include e.g. NAS [Kim et al, 2022] or DCLS [3]. Starting with our result, one could study how input partitioning influences the training dynamics/performance, e.g., is it advantageous to initialize networks with a dense and complex baseline input partitioning as in ANNs [2]? Convergence speed and generalization in SNNs could also benefit from this strategy.

[1]: Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective, Chen, Gong, Wang, ICLR 2021

[2] Compelling ReLU Networks to Exhibit Exponentially Many Linear Regions at Initialization and During Training, Milkert, Hyde, Laine, arxiv 2025

[3] Learning Delays in Spiking Neural Networks using Dilated Convolutions with Learnable Spacings, Hammouamri, Khalfaoui-Hassani, Masquelier, ICLR 2024

2. Continuous- vs discrete-time SNN expressivity

• [Neuman et al 2024] shows that shallow continuous-time SRM-SNNs with temporal coding and linear response function are universal approximators for compactly supported continuous functions. They restrict to single-spike neurons and does not quantify network size.

• [Singh et al 2024; Stanojevic et al 2024] show the ability of similar models in emulating ReLU networks, possibly requiring larger architectures. Due to known results for ReLU networks, one can deduce expressivity results for these SNNs.

• [Comsa et al 2020] shows universality of similar SNNs but with exponential response function. Notably, target function outputs must be bounded below. For $\epsilon$-accuracy, they require $\Theta(n(\Gamma\sqrt{n}/\epsilon)^n)$ neurons, compared to our $\Theta((\Gamma/\epsilon)^n)$.

• For input partitioning, [Singh et al 2024] show that single-output single-layer continuous-time SNNs with linear response realize piecewise linear functions with at most $2^n-1$ pieces. In our case, this bound becomes $(T^2+T+2)/2$ constant regions, independent of input dimension.

However, such direct comparisons of complexity across models may be not entirely fair, as different models realize distinct function classes.

3. Partitioning complexity with temporal data

Extending our result to temporal data is challenging, as defining input partitioning is non-trivial. A natural approach is to stack temporal elements and repeat our analysis. While feasible, it is unclear if this approach is meaningful, as the simple stacking process may overlook temporal dependencies. Also, we need to characterize what functions are realized to get a reasonable complexity proxy. Since the output for dynamic data is not necessarily constant, we must reconsider the notion of constant regions.

For now, we analyze static data as a first step toward that goal. As our experiments aim to validate our theoretical results, which do not cover dynamic data, experiments with dynamic data are beyond this paper’s scope.

4. Quadratic growth of partitioning complexity

With growing latency, both expressivity and computational complexity increase. This is reflected by practical observations that increasing $T$ leads to performance gains, but inference and training become more expensive in terms of energy and time consumption. Therefore, investigating the trade-off between computational and model complexity is important. The quadratic growth of the input partitioning complexity in latency suggests that expanding SNNs in temporal domain may increase the model complexity, but not at an exponential pace. Moreover, Section 5.2 suggests that the constant regions might be very thin with growing latency. Hence, one could prefer a smarter way to scale up and improve performance, rather than simply increasing the latency. This also aligns with the current trend of designing low-latency SNNs.

Counting regions is just the first step, as it does not provide sufficient practical insights. In classical DL, follow-up studies have been published on various aspects such as shape, size, and evolution over time, which offer valuable practical implications (see Related Works). Similarly, for spiking neural networks (SNNs), it is important to explore these aspects, along with their temporal dependencies.

We thank Reviewer QUDk for the feedback. Below, we address the mentioned concerns individually:

1. Relationship to the work [1].

From a high level, this work discusses the capacity of neurons under a specific model with time encodings. The capacity in their work is related to the ability of the neuron to learn time-encoded spatio-temporal patterns. This has several key differences with our work, as we list below:

(1) Model: The model considered in [1] is a spike-response model with time encoding, which differs from ours in several key aspects. Their approach primarily focuses on a specific time-based encoding for spatio-temporal patterns. In contrast, in the model we consider, time plays a key role in the neuronal dynamics but not in the encoding scheme. Our framework does not rely on a specific encoding strategy tailored to particular data. Instead, it adopts a more general representation of spike patterns, consistent with the use of SNNs as computational models—a perspective reinforced by the widespread adoption of the discrete LIF model in diverse practical applications. Moreover, the methodology in [1] is limited to a single neuron, and therefore does not incorporate the concept of a neural network.

(2) Capacity measure: Their measure of capacity appears to be linked to the neuron's ability to learn specific patterns and is therefore closely tied to the learning rule. In contrast, our work does not theoretically address the training process. Instead, our capacity measures are based on function approximation and the study of linear regions. These notions seem largely unrelated to those developed in [1].

(3) Motivation: The primary motivation in [1] is to understand the learning process in biological neurons through a spiking neuron model with biologically plausible learning. In contrast, our work focuses on the discrete LIF model, treating it as a computational framework. Although this model is widely used, many of its theoretical properties remain insufficiently explored. Our work aims to contribute in that direction.

In summary, our work differs from [1] in its motivation, modeling framework, and capacity measures.

2. Generalization to other neuron models.

First, it should be mentioned that the elementary LIF neuron model discussed in our paper is covered by many more complex (and often more biologically plausible) neuron models in the literature as a special case. For instance, the current-based model (see e.g. [Eshraghian et al., 2023]) can be reduced to the LIF model by setting the synaptic current decay rate to $1$. In this case, our universality result can be directly extended to these neuron models. Certainly, SNNs may gain some benefit in terms of expressive power when being constructed based on a more complex neuron model, which may lead to better approximation rates for certain function classes.

Second, concerning the input partitioning result, the relevant proofs rely mainly on the affine linearity of the synaptic response (given by the weight matrices and biases) as well as the Heaviside spike activation, which are the common trends in SNNs. Thus, we think that our technique might also apply to other neuron models, yet with several technical modifications. As an example, one may consider the current-based model mentioned above. By taking the sum over time steps, one again obtains a time-dependent affine relation between the membrane potential over the first hidden layer and the input vector, which would lead to an analogous input partitioning as in the LIF case.

We would also like to clarify that our focus on the basic LIF model is not a limitation but rather because of its widespread use, simplicity, and computational efficiency. While extending our results to other neuron models, as discussed above, is realistic and beneficial, a more rigorous exploration would require careful consideration, and we leave this for future work.

Finally, our discussion above relies on the time discretization as an essential high-level aspect in the model comparison. We are not aware if the reviewer counts this into the neuron model (i.e. if you are asking for generalization of our results to continuous-time models). A continuous-time model would likely differ crucially from our discrete-time model (please refer to our related works section and also our response to Reviewer V5so), and hence it is not clear to us how to meaningfully generalize our results to continuous time.

[1]: The tempotron: a neuron that learns spike timing–based decisions. R. Güter and H. Sompolinsky, Nature Neuroscience 9, 420–428, 2006