Accelerating Linear Recurrent Neural Networks for the Edge with Unstructured Sparsity

We gratefully thank the reviewer for their effort and feedback. We would like to address their questions, and we’d appreciate any raise in score if our arguments convince them. We will be happy to hear and address any further feedback.

Generalization to different architectures: preliminary results on HGRN

To expand our results to different linear RNN architectures beyond S5, we can present some preliminary results on training a 370M parameter language model based on the HGRN architecture [1]. Using the same iterative magnitude pruning setup as described in our paper, we trained the HGRN model on the FineWeb dataset for ~5B tokens with different target sparsity ratios from 0% to 90%. The training loss for the 90% sparse model is ~9% higher than for the dense model, see the Table below. Additional details are available in the Figure available at https://figshare.com/s/a8cd30c07af56f4ddbbf. These findings confirm a similar trend as what we reported in our paper on real-time sequence modeling tasks. More work is underway to scale up the language modeling experiments, and verify their validity on real-world datasets and benchmarks.

Ablation study on quantization schemes and sparsity levels

We agree with the reviewer that exploring the large design space of quantization schemes and sparsity distributions would provide additional insights in this domain. On quantization, previous work on S5 [2] showed that, compared to W8A8, lower precision significantly impacted accuracy. Since Loihi’s message format doesn’t provide any performance advantage for 8-bit versus 16-bit activations, we selected W8A16 which preserves higher accuracy. Regarding sparsity, we selected a 90% target after some initial experimentation, as it resulted in a good trade-off between impact on task accuracy and MACs reduction. We could add an ablation study on sparsity levels to Appendix A, if the Reviewer finds it helpful for readers. We would also acknowledge these directions for future work by updating the last sentences of Section 4 with the following:

Finally, further exploration of quantization schemes and sparsification techniques could offer deeper insights into optimal model design for different hardware platforms. In particular, leveraging advanced data types for quantization (e.g., FP8) and adopting a more fine-grained selection of sparsity levels, potentially guided by iterative hardware profiling, are promising directions for future research.

References

1. Zhen Qin, Songlin Yang, Yiran Zhong, “Hierarchically Gated Recurrent Neural Network for Sequence Modeling”, NeurIPS 2023 Spotlight.
2. Abreu, Steven, et al. "Q-S5: Towards quantized state space models." arXiv preprint arXiv:2406.09477 (2024).

We gratefully thank the reviewer for their effort and feedback. We would like to address their questions, and we’d appreciate any raise in score if our arguments convince them. We will be happy to hear and address any further feedback.

Clarification on Figure 6

We regret the suboptimal presentation of Figure 6 and appreciate the reviewer’s nudge to clarify. We confirm that FPX is a typo and should read FXP, for fixed-point. We will replace the caption with the following in the camera-ready version:

Base: trained a 32-bit floating-point model and applied post-training quantization without additional quantization-aware fine-tuning. QAT: trained the model with quantization-aware training using W8A16 quantization for the forward pass, as straight-through estimators for the backward pass. The results show that the Base model without QAT performs slightly better in FP32 than the QAT model, but significantly worse in static quantization and fixed-point precision. The model shown here is the sparse-6 variant, see Figure 4.

Previous work on deployment of linear RNNs to neuromorphic devices

Since the now broad interest in linear RNNs is relatively recent, there is limited previous work on deployment of this architecture to neuromorphic devices, and, to the best of our knowledge, no previous work on hardware-aware model compression in this context. Previous work on Loihi 2 demonstrated the implementation of S4D [1], a state space model based on the Legendre Delay Network [2], and a MatMul-free LLM [3]. While S4D could potentially target the same application domains as S5, Ref. [1] only reports benchmarks on simple synthetic tasks (sMNIST and sCIFAR) and without model compression. For this reason, a direct performance comparison is not possible. Ref. [2] implemented a state space model with a spiking neural network on a neuromorphic processor – also without model compression techniques. The MatMul-free LLM in Ref. [3] uses ternary weight matrices for model compression but doesn’t leverage sparsity advantages from the neuromorphic chip on which it is deployed, thereby making the comparison to our present work difficult. We are not aware of any further deployment of linear RNNs on neuromorphic hardware.

References

1. Meyer, Svea Marie, et al. "A Diagonal Structured State Space Model on Loihi 2 for Efficient Streaming Sequence Processing." arXiv preprint arXiv:2409.15022 (2024).
2. Gaurav, Ramashish, Terrence C. Stewart, and Yang Yi. "Legendre-SNN on Loihi-2: Evaluation and Insights." NeurIPS 2024 Workshop Machine Learning with new Compute Paradigms.
3. Abreu, Steven, et al. "Neuromorphic Principles for Efficient Large Language Models on Intel Loihi 2." arXiv preprint arXiv:2503.18002 (2025).

We gratefully thank the reviewer for their effort and feedback. We would like to address their questions, and we’d appreciate any raise in score if our arguments convince them. We will be happy to hear and address any further feedback.

Generalization to different accelerators

Please refer to our response to Reviewer VTiJ.

Generalization to different tasks

Please refer to our response to Reviewer VTiJ.

Impact of batch processing

We appreciate the reviewer’s question regarding the impact of batching on our benchmarking results. We address it in the following paragraph, which we plan to include in Section 3.3 of the camera-ready version.

Although edge applications are typically thought of as single-batch applications, some workloads at the edge require small-batch inference, e.g., de-noising audio streams from multiple on-device microphones. For this reason, it is important to investigate how batch processing affects latency and energy efficiency for the two hardware architectures. While Intel Loihi 2 doesn’t natively support batching in the sense of processing multiple independent samples though the same model instantiation, the parallel inference of independent sequences can be obtained by replicating the model on the chip as many times as required by batch size. We extended the results in Table 1 to compare the effect of this implementation of batching to the usual batch processing of the Jetson Orin GPU. Figure X (available at https://figshare.com/s/4d6035c9a2c4739d7201) shows total latency and energy per sample across batch sizes, from 1 to 32. The results demonstrate that our approach on Loihi 2 maintains its large latency advantage while the energy efficiency gain is only maintained for the small-batch regime (below 8 samples), which is typical of edge inference applications. As expected, batch processing improves energy efficiency for the GPU, since the cost of data movement associated with loading the model is offset by the parallelized evaluation of multiple samples.

Novelty and originality of the work

We thank the reviewer for pointing out that we missed to express the novelty and originality of our work. We also appreciate their statement that our work “can be a good reference for the deployment of linear RNN models on edge devices”. In addition, we would argue that our paper has two further main original/novel contributions.

First, this paper spearheads the idea that neuromorphic processors are an ideal platform for the emerging class of linear RNNs. This is particularly due to the tight integration of massively parallel compute and memory in neuromorphic hardware, which can efficiently update stateful recurrent neurons. Neuromorphic processors are typically designed for low-latency processing of sequentially incoming sensory signals, and can thus particularly benefit from the advantageous scaling trends of linear RNNs with long sequences. Our present paper is the first peer-reviewed publication that explores the combination of model compression techniques required to exploit the synergy between neuromorphic processors and linear RNNs.

Second, while we agree that we applied off-the-shelf compression techniques, we would argue that our paper combines them into a novel training recipe necessary to leverage the specific features typical for neuromorphic computing for an optimized deployment of linear RNNs. We would like to kindly point out that, according to the 2025 ICML reviewer guidelines, “Originality need not mean wholly novel methods. It may mean a novel combination of existing methods to solve the task at hand, a novel dataset, or a new way of framing tasks or evaluating performance so as to match the needs of the user.” Our new training and deployment recipe has proven to bring tangible performance and energy advantages on real hardware for a real-world task that requires low-latency, low-power execution, audio denoising. We will release this new training pipeline upon acceptance so that the community can extend it or apply it to additional applications.

We will update the introduction in the camera-ready version accordingly.

We gratefully thank the reviewer for their effort and feedback, and we would like to address their questions. We will be happy to hear and address any further feedback.

Generalization to different accelerators

Reviewers VTiJ and jRFp raised the need for a discussion about the generalizability of our methodology and benchmarking to other hardware platforms (e.g., SpiNNaker 2 and IBM NorthPole). While we adopted a hardware-aware approach focused on Loihi 2, we believe that platforms with similar feature sets can benefit from it. However, since most neuromorphic processors are currently research prototypes, they don’t provide public access (like IBM NorthPole) and/or lack a high-level programming framework that would allow the transfer of models from one platform to another. Following the approach proposed in NeuroBench [1], we will publicly release our training pipeline and checkpoints to enable the implementation on other neuromorphic processors by the relevant experts.

We further address the generalizability discussion in the following paragraph, which we plan to add to Section 4 in the camera-ready version.

While the proposed hardware-aware methodology is tailored to leverage the Loihi-specific feature set, we believe that platforms with similar characteristics could potentially benefit from the results we presented. Neuromorphic processors such as SpiNNaker 2 [2] and IBM NorthPole [3], which presents similar architecture patterns to Loihi 2, would greatly benefit from our methodology on activation and weight sparsity. In addition, other platforms with tight compute-memory integration, such as Cerebras Wafer-Scale Engine (WSE-3) [4], provide support for unstructured sparsity, even if targeting datacenter-scale applications.

Generalization to different tasks

Reviewers VTiJ and jRFp noted that extending our benchmarking to other datasets would strengthen our claims and the relevance of our work for the broader edge inference community. Starting from the S5 baseline [5] and following our methodology, we are running experiments on the keyword spotting SpeechCommands V2-35 (SC35) dataset [6], which provides a common real-time use case for the edge. The preliminary results, available at https://figshare.com/s/3d0ba8e3f515535871ba, compare the training curves for a narrow dense and wider sparse model (90% sparse weights and ReLU activations) with similar inference MACs. The plot shows that the sparse model (still under training) is on track to match or exceed the accuracy of the dense counterpart. Similar experiments are underway across inference compute budgets, and we plan to add such results in the same form as Figure 4 to the camera-ready version. Since our methodology seems to generalize without modification to SC35, we expect it to also generalize to other sequence modeling applications commonly solved with linear RNNs, such as bio-marker signal monitoring [7], time series forecasting [8], or action recognition [9].

References

1. Yik, Jason, et al. "The neurobench framework for benchmarking neuromorphic computing algorithms and systems." Nature Communications 16.1 (2025): 1545.
2. Mayr, Christian, Sebastian Hoeppner, and Steve Furber. "SpiNNaker 2: A 10 million core processor system for brain simulation and machine learning-keynote presentation." Communicating Process Architectures 2017 & 2018. IOS Press, 2019. 277-280.
3. Modha, Dharmendra S., et al. "Neural inference at the frontier of energy, space, and time." Science 382.6668 (2023): 329-335.
4. Lie, Sean. "Cerebras architecture deep dive: First look inside the hardware/software co-design for deep learning." IEEE Micro 43.3 (2023): 18-30.
5. Smith, Jimmy TH, Andrew Warrington, and Scott W. Linderman. "Simplified state space layers for sequence modeling." arXiv preprint arXiv:2208.04933 (2022).
6. Warden, Pete. "Speech commands: A dataset for limited-vocabulary speech recognition." arXiv preprint arXiv:1804.03209 (2018).
7. Pimentel, Marco AF, et al. "Toward a robust estimation of respiratory rate from pulse oximeters." IEEE Transactions on Biomedical Engineering 64.8 (2016): 1914-1923.
8. Schirmer, Mona, et al. "Modeling irregular time series with continuous recurrent units." International conference on machine learning. PMLR, 2022.
9. Kuehne, Hildegard, et al. "HMDB: a large video database for human motion recognition." 2011 International conference on computer vision. IEEE, 2011.