Spiking Hybrid Attentive Mechanism with Decoupled Layer Normalization for Joint Sound Localization and Classification

Comprehensive analysis of the robustness to noisy environment is insufficient.

R: Here, we evaluate the noise robustness of our model under various noisy conditions. We select the well-trained SpikSLC-Net-2-128 on SLoClas dataset as the testing model. The noise sounds are from SLoClas dataset, which contains the recordings of 6 types of noise at 4 DoAs. We randomly select the noise audio from one of the four directions, 0°, 90°, 180°, 270°. We regard the noise as the addictive background noise. In order to simulate the background noise from all directions, we add the noise sound clip to microphone signal in the channel dimension. In Tab. R4-2, we show the results on noisy data of various Signal-to-Noise ratio (SNR). It is understandable that MAE decreases as SNR increases across all conditions. For high SNR test data, our model performs reasonably well.

As in [1], comparison of computation complexity and estimated power consumption are highly recommended to be embraced in the experiments.

R: The contributions of previous SNN methods focus on coding methods rather than models, so it is difficult to make a fair comparison of algorithm complexity. Furthermore, other methods cannot perform both sound source localization and recognition simultaneously. Thus, we only provide the computation complexity and estimated power consumption of our models and other ANN models. In SNNs, the synaptic operations (SOPs) are used to describe the computation complexity, which is calculated by

$\operatorname{SOPs}(l)=fr \times T \times \operatorname{FLOPs}(l)$,

where $l$ is a layer in SpikSLC-Net and $fr$ represents the firing rate of the input spike train, and $T$ denotes the time step. $\operatorname{FLOPs}(l)$ refers to floating point operations of $l$, which is the number of multiply-and-accumulate (MAC) operations. And SOPs is the number of spike-based accumulate (AC) operations. Refer to previous works [1][3], we assume that the operations are implemented on the 45nm hardware [4], where the theoretical energy consumption $EMAC = 4.6pJ$ and $EAC = 0.9pJ$.

The results are shown in Tab. R4-3. From the results, it can be seen that the OPs and the theoretical energy consumption of our models are much lower compared with ANN models that have good performance.

We will add these promising comparison results to the revised version.

Reference:

[1] Zhaokun Zhou, Yuesheng Zhu, Chao He, Yaowei Wang, Shuicheng Yan, Yonghong Tian, and Li Yuan. Spikformer: When spiking neural network meets transformer. arXiv preprint arXiv:2209.15425, 2022.

[2] Cao, Yin and Iqbal, Turab and Kong, Qiuqiang and An, Fengyan and Wang, Wenwu and Plumbley, Mark D. An improved event-independent network for polyphonic sound event localization and detection. ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[3] Man Yao, Guangshe Zhao, Hengyu Zhang, Yifan Hu, Lei Deng, Yonghong Tian, Bo Xu, and Guoqi Li. Attention spiking neural networks. arXiv preprint arXiv:2209.13929, 2022.

[4] Mark Horowitz. 1.1 computing’s energy problem (and what we can do about it). In 2014 IEEE International Solid-State Circuits Conference Digest of Technical Papers (ISSCC), pages 10–14. IEEE, 2014.

Thanks for your constructive and valuable feedback. We would like to address your concerns below.

Though achieving state-of-the-art performance on SLoClas dataset, the proposed architecture still lacks novelty. According to my understanding, the core part of the architecture is the SHAF block, however its components are just copies or slight modifications/extensions of spiking self attention mechanism proposed in [1]. The extension seems straight forward and may not be regarded as a genuine technical contribution.

R: While the SCA module in SHAF block is inspired by SSA module proposed in [1], their input types and functionalities are completely different. The SSA, based on the spiking self-attention mechanism, is utilized to model long-range interdependencies within the same input. But the SCA applies the spiking cross-attention mechanism to align features between different input stream. To the best of our knowledge, this is the first work to explore the effectiveness of cross-attention mechanism in SNNs. We believe that this investigation not only promotes the application of SNN in the audio field, but also paves the way for further research on multimodal SNN models.

Moreover, the whole SHAF block employs a hybrid attention mechanism, instead of a single self-attention mechanism or a single cross-attention mechanism. The block has strong ability in learning intra-modal acoustic temporal dependencies and aligning inter-modal acoustic relationships. Each component of the block has a significant impact on the model performance.

In addition, the DSLN method we proposed is also a core part of the architecture. Different from vanilla LayerNorm, DSLN only calculates the mean while keeping the variance fixed during inference. And we apply a re-parameterization technique to fold the trained DSLN into the weights so that no additional computation and energy consumption are generated.

Some of the notations are not align with the others. For example, sometimes the embedding feature dimension is denoted as, but sometimes not (Fig. 2).

R: We will correct them in the revised version, thank you for your help!

Though performing best on SLoClas dataset, it is hard to conclude that the current work is better than previous works in terms of sound localization and classification. More experiments on various datasets is recommended.

R: Since there are limited datasets and models for joint sound localization and classification, we construct a suitable dataset named DCASE 2019 Part. The dataset is the part of the DCASE 2019 dataset without overlapping sound sources. It has 11 sound sources and the azimuth angle interval is 10 degrees. Considering previous SNN methods have not made their code available, we resort to using traditional ANN methods for comparison purposes. Specifically, we utilize a modified version of EINV2 [2] which is originally designed for sound event detection and implement an ANN version of our model (SLC-Net). When evaluating models on DCASE 2019 Part dataset, the threshold for accuracy calculation is set at 5 degrees, and it is set to 2.5 degrees on SLoClas dataset. The results are shown in Tab. R4-1. As we can see, our method outperforms other methods on both sound event localization and classification tasks.

Lack of clarity in methods.

R: Thanks for your carefully reading. We will align the symbols and add more detailed clarifications in the revised version.

In Fig. 2, $N$, $F$, $B_g$ and $B_m$ represent the the total microphone pairs number, the frames number, the feature dimension of each frame in GCC-PHAT and the feature dimension of each frame in log-Mel spectrogram, respectively. In Eq. 8, the symbols are general, where $N$ is equal to $F$ in Fig. 2 and $D$ is equal to $E$ in Fig. 2.

In the dimensions $T \times N \times D$, $T$ denotes the time step, $N$ denotes the frames number, and $D$ denotes the embedding dimension. The attention operation indeed extends over time step. The dimensions of $W_Q$, $W_K$ and $W_V$ are $D \times D$. The product $X_{\alpha} W_Q$ means projecting each frame of data at each time step to embedding space. According to [1], the batch norm around this product can normalize the embeddings and prevent neurons from firing dense spikes. $\operatorname{SN(.)}$ represents the LIF neuron layers and the dimensions of input and output are both $T \times N \times D$.

In Eq. 15, $W$ represents the parameters of the previous Linear layer. $m$ and $n$ denote the input channel dimension and output channel dimension, respectively. $A_i$ and $W'_i$ are the expressions of constants that result from previous calculations.

Reference:

[1] Zhaokun Zhou, Yuesheng Zhu, Chao He, Yaowei Wang, Shuicheng Yan, Yonghong Tian, and Li Yuan. Spikformer: When spiking neural network meets transformer. arXiv preprint arXiv:2209.15425, 2022.

[2] Cao, Yin and Iqbal, Turab and Kong, Qiuqiang and An, Fengyan and Wang, Wenwu and Plumbley, Mark D. An improved event-independent network for polyphonic sound event localization and detection. ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[3] Man Yao, Guangshe Zhao, Hengyu Zhang, Yifan Hu, Lei Deng, Yonghong Tian, Bo Xu, and Guoqi Li. Attention spiking neural networks. arXiv preprint arXiv:2209.13929, 2022.

[4] Mark Horowitz. 1.1 computing’s energy problem (and what we can do about it). In 2014 IEEE International Solid-State Circuits Conference Digest of Technical Papers (ISSCC), pages 10–14. IEEE, 2014.

Lack of strong baselines.

R: Our framework is based on full SNNs, so it’s reasonable to compare our model with other SOTA SNN methods on the same datasets they used. The previous best ANN method for sound localization and classification on SLoClas dataset is GCC-PHAT-CNN. In order to demonstrate the superiority of our model, we utilize a modified version of EINV2 [2] which is originally designed for sound event detection and implement an ANN version of our model (SLC-Net) for additional experiments. The corresponding results are shown in Tab. R3-3. The advantage of our model in DoA estimation is very obvious. According to the characteristics of SNN, we think that it is very possible for SNN to outperform ANN in specific cases. Additionally, it is plausible that the performance of the current best ANN model is still insufficient.

Narrow evaluation.

R: Here, we conduct more experiments on different datasets and analyze the computation complexity and estimated power consumption to show the superiority of our SNN approach on two tasks. Since there are limited datasets and models for joint sound localization and classification, we construct a suitable dataset named DCASE 2019 Part. The dataset is the part of the DCASE 2019 dataset without overlapping sound sources. It has 11 sound sources and the azimuth angle interval is 10 degrees. When evaluating models on DCASE 2019 Part dataset, the threshold for accuracy calculation is set at 5 degrees, and it is set to 2.5 degrees on SLoClas dataset.

In SNNs, the synaptic operations (SOPs) are used to describe the computation complexity, which is calculated by

$\operatorname{SOPs}(l)=fr \times T \times \operatorname{FLOPs}(l)$,

Where $l$ is a layer in SpikSLC-Net and $fr$ represents the firing rate of the input spike train, and $T$ denotes the time step. $\operatorname{FLOPs}(l)$ refers to floating point operations of $l$, which is the number of multiply-and-accumulate (MAC) operations. And SOPs is the number of spike-based accumulate (AC) operations. Refer to previous works [1][3], we assume that the operations are implemented on the 45nm hardware[4], where the theoretical energy consumption $EMAC = 4.6pJ$ and $EAC = 0.9pJ$.

The results are shown in Tab. R3-4 and Tab. R3-5. The number of operations (OPs) refers to SOPs in SNN and FLOPs in ANN.

As we can see, our models achieve the best performance while maintaining the lowest energy consumption compared with ANN models.

We appreciate your detailed comments. We would like to address your concerns and questions below.

Why this combination of problems and approaches?

R: Our research primarily focuses on the domains of spiking neural networks and audio. The motivation of our work has been illustrated in Introduction section of our manuscript and we will give more discussion about the reason that we use SNNs in joint sound localization and classification.

Firstly, as we know, SNN has the characteristics of bio-interpretability, lower power consumption, higher robustness, and better ability to extract spatio-temporal features. So it provides a more energy-efficient and bio-plausible way to perform such tasks in the audio field.

Secondly, for humans, localizing and identifying sound sources simultaneously through binaural cues is a crucial and basic ability. Therefore, it’s very meaningful to investigate the information sharing ability of brain-inspired SNNs from the SELC task.

Thirdly, previous work about SNNs has largely focused on the computer vision and the performance of SNN in handling other modal data has not been fully explored. Different from images, audio data has a temporal dimension and is more sparse, posing new challenges for SNN architectures. For example, how to integrate different acoustic features, how to normalize features at the frame level, and how to design low-power networks for audio.

Finally, if we want to employ SNN in sound localization and classification, we have to solve the above basic problems in the specific task first. LN is a widely used normalization method to process temporal data like audio in ANNs. However, the vanilla LN do not conform to the calculation characteristics of SNNs, since it needs to be calculated dynamically during runtime and involves a substantial number of floating-point operations. Therefore, it’s very essential to develop a suitable LN method for SNNs.

Claims not supported well by experiments and evidence.

R: Multi-task learning. Due to the powerful performance of our model and the initial high accuracy of sound source classification, multitask learning has limited impact on further improving sound source classification accuracy. Instead, it focuses more on enhancing sound source localization accuracy. Our results indicate that multitask learning is feasible in SNNs, which will also help promote the development of SNNs.

SHAF mechanism. The SHAF mechanism consists of spike-form cross-attention and self-attention, which is able to integrate acoustic features. In SNNs, the direct input of LIF neurons can be real-valued [1]. The float-point $Q,K,V$ refers to the results of $XW$, serving as the input of LIF neurons. So it can be deployed on neuromorphic hardware.

DSLN method. Our goal is to fold the LN calculation process into other layers like FC while retaining as many statistics as possible so that we can eliminate the time cost and computational energy consumption of DSLN. In practice, we choose to use a fixed variance and dynamically calculate the mean in the stage of inference. The re-parameterization procedure are derived in Eq. 15.

Here, we conduct more ablation experiments on SHAF and DSLN to demonstrate the effectiveness of our method. For the SHAF, we apply SpikSLC-Net-2-128 and the model has two FC layers in each head. For the DSLN, to avoid the impact of head design on model performance, we apply SpikSLC-Net-3-128 with lightweight heads which are equipped with only one FC layer. The results are listed in Tab. R3-1 and Tab. R3-2. As we can see, their improvement in model performance is mainly reflected in the DoA estimation task, since the sound localization is more challenging than classification. The SCA and SSA modules in SHAF both can greatly improve the performance of the model. And the DSLN module can effectively alleviate overfitting in larger model.

Thank you for your recognition of our work. And now we will address your concerns in the following.

Do you have any other results to compare against traditional ANN methods?

R: Since there are limited datasets and models for joint sound localization and classification, we construct a suitable dataset named DCASE 2019 Part. The dataset is the part of the DCASE 2019 dataset without overlapping sound sources. It has 11 sound sources and the azimuth angle interval is 10 degrees. For traditional ANN methods, we employ a modified version of EINV2 [1] which is originally used in sound event localization and detection and implement an ANN version of our model (SLC-Net). When evaluating models on DCASE 2019 Part dataset, the threshold for accuracy calculation is set at 5 degrees, and it is set to 2.5 degrees on SLoClas dataset. The results are shown in Tab. R2-1. As we can see, our method outperforms the ANN methods on both sound event localization and classification tasks.

Reference:

[1] Cao, Yin and Iqbal, Turab and Kong, Qiuqiang and An, Fengyan and Wang, Wenwu and Plumbley, Mark D. An improved event-independent network for polyphonic sound event localization and detection. ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

Thank you for your constructive comments and suggestions for improvement. We would like to address your concerns and answer your questions in the following.

Following above in weaknesses, I think it would be helpful to perform ablation studies, showing the role of the proposed SHAF blocks and DSLN specifically in the superior performance.

R: We perform more ablation studies on SHAF and DSLN modules to show their significance in our models. For the SHAF experiment, we apply SpikSLC-Net-2-128 and the model has two FC layers in each head. For the DSLN experiment, to avoid the impact of head design on model performance, we apply lightweight heads which are equipped with only one FC layer. The results are listed in Tab. R1-1 and Tab. R1-2. As we can see, their improvement in model performance is mainly reflected in the DoA estimation task, since the sound localization is more challenging than classification. The SCA and SSA modules in SHAF both can greatly improve the performance of the model. Besides, the DSLN module can effectively alleviate overfitting in larger model.

Can authors provide benchmarking with F1 score, precision, recall?

R: In order to fully demonstrate the superiority of our model, we select the part of the DCASE 2019 dataset without overlapping sound sources for additional experiments (DCASE 2019 Part). It has 11 sound sources. What’s more, we implement an ANN version of our model (SLC-Net) for comparison on SLoClas dataset and DCASE 2019 Part dataset. The corresponding metrics are shown in Tab. R1-3. For DCASE 2019 Part, the threshold for accuracy calculation is set at 5 degrees, as the azimuth angle interval is 10 degrees. We can find that our method displays absence of potential bias in sound classes, ensuring unbiased precision and recall measurements.

Can authors provide an algorithmic complexity analysis of this SNN vs. previous SNN?

R: Since previous SNN methods do not release their code and their contributions focus on coding methods, it is difficult to make a fair comparison of algorithm complexity. Moreover, it is worth noting that other methods are unable to perform both sound source localization and recognition simultaneously, as we do. Here, we present the algorithmic complexity analysis of our models and other ANN models on SLoClas dataset in Tab. R1-4. The number of operations (OPs) refers to synaptic operations (SOPs) in SNN and FLOPs in ANN. It can be seen that our method achieves high accuracy while requiring the lowest algorithmic complexity.