SpGesture: Source-Free Domain-adaptive sEMG-based Gesture Recognition with Jaccard Attentive Spiking Neural Network

To reviewer Thank you for your thorough review and insightful feedback on our work. We are delighted that you recognize the strengths of our proposed Jaccard-based attention mechanism, our SNN-oriented SFDA algorithm, and the high accuracy (89.26%) achieved on our newly collected sEMG gesture dataset. We are particularly pleased that you appreciate our focus on real-world application experiences and deployments, which we believe are crucial for advancing human-computer interaction. We value your suggestions for improvement and will consider them carefully to enhance the soundness and presentation of our work. Your feedback is instrumental in guiding our future research efforts.

About Jaccard attention reflects 'attention' within tokens and intuitive explanation. Thank you for your insightful question. I appreciate the opportunity to provide a more detailed explanation of how our proposed Jaccard-based attention mechanism (SJA) reflects "attention" within tokens.

In traditional attention mechanisms, the relationships between the query (Q), key (K), and value (V) matrices are established through dot product operations. These dot products are then normalized and used to weigh the values in the value matrix (V). The essence of this process is to compute the similarity between each query vector and all key vectors, using these similarities to allocate attention weights and ultimately obtain a weighted sum of the values.

Our SJA mechanism replaces the dot product similarity with the Jaccard similarity, which is more suited for the binary nature of Spiking Neural Network (SNN) outputs. This is particularly useful as SNN data is often sparse and binary.

The SJA mechanism is defined by the following formula:

$

\mathrm{SJA}\left(\mathbf{Q}, \mathbf{K}\right) = \frac{\sum_{ij}\min\left(q_{ij}, k_{ij}\right)}{\sum_{ij}\max\left(q_{ij}, k_{ij}\right) + \epsilon} \ \mathbf{V},

$ are the corresponding elements in the query and key matrices.

Intuitive Explanation of SJA's Effectiveness: Imagine each token's spike train as a binary pattern representing different features or characteristics. The Jaccard similarity helps identify how much overlap (or commonality) there is between the features of the two tokens. By focusing on the overlapping features, the model can better understand the importance of these features and adjust the values (from $V$) accordingly, enhancing the overall representation and prediction accuracy. In detail,

1. Utilizing Sparsity: SNN outputs are typically sparse, and SJA leverages this sparsity by focusing on non-zero elements. Traditional dot product methods can be inefficient for sparse data as they involve numerous zero-value multiplications. SJA, however, uses element-wise minimum and maximum operations, concentrating on the non-zero elements, thus reducing computation time and energy consumption.
2. Element-wise Similarity Measurement: The Jaccard similarity measures the intersection ratio to the union of two sets. For binary vectors, this effectively captures the proportion of shared active elements. In our formula, $\min(q_{ij}, k_{ij})$ and $\max(q_{ij}, k_{ij})$ represent the intersection and union of corresponding elements, respectively. This method directly reflects the similarity between the query and key vectors at the same positions, providing a meaningful measure of attention for sparse binary data.
3. Improved Computational Efficiency: Using element-wise minimum and maximum operations instead of matrix dot products, SJA significantly reduces computational complexity. Traditional attention mechanisms have a complexity of $O(n^2 \cdot d)$, whereas SJA has a complexity of $O(b)$, where $b$ is the number of non-zero elements. This reduction in complexity is particularly beneficial for handling large-scale sparse data, making SJA more efficient and suitable for deployment on spiking chips.

In summary, the SJA mechanism captures attention by measuring the Jaccard similarity between the query and key vectors, focusing on the significant non-zero elements, and leveraging the sparsity of SNN outputs. This approach not only reduces computational complexity but also enhances energy efficiency, making it an effective and efficient method for attention in SNNs.I hope this explanation clarifies how SJA works and why it is effective. Thank you for your attention and for providing the opportunity to elaborate on our work.

About algorithm time and memory usage for the SJA. Thank you for your valuable feedback. We appreciate your suggestion to clarify the comparison of benefits in Section 5.5. We apologize for any confusion caused by our description. In Section 5.5, specifically in Figure 5, we indeed present the comparison of inference speed and RAM usage between SJA, Efficient Attention, and RAW Attention. The data shown pertains solely to the attention part of the algorithm, not the entire algorithm. Thank you for highlighting this area for improvement. We will ensure that this is more clearly communicated in the revised version of our paper.

To reviewer: Thank you for your thorough and insightful review of our paper. We are delighted that you found our contributions noteworthy. We appreciate your recognition of the Jaccard Attention SNN model, which enhances feature representation while maintaining computational efficiency and is validated through ablation experiments. We are also grateful for your acknowledgment of our innovative introduction of source-free domain adaptation in SNNs, leveraging membrane potential as a memory feature to improve generalization performance. Your positive remarks on our systematic comparison of SpGesture’s performance and the collection and open-sourcing of the multi-posture sEMG dataset further encourage us. Thank you for your valuable feedback and suggestions, which will guide our future research. Below, we will specifically address your questions.

About the design of JASNN, its consideration of other similarity measures, and the scalability of Jaccard attention. Thank you for this insightful question. Indeed, we did consider other similarity measures during the development of JASNN. We experimented with cosine similarity and Pearson correlation coefficient but found that Jaccard similarity performed best for binary sparse data, which aligns well with the spiking nature of SNNs. Regarding the application to other SNN types, we have successfully applied Jaccard attention to other SNN architectures, including LSNN (Long Short-Term Memory Spiking Neural Networks). We observed consistent performance improvements across different SNN types, indicating the generalizability of our approach. In the revised version, we will include these additional results to demonstrate the broader applicability of our method.

About the characteristics of the sEMG distribution shifts corresponding to different forearm postures. The characteristics of the sEMG distribution shifts corresponding to different forearm postures are demonstrated in Appendix Figure 9. We used data from Posture 1 for inference on data from Postures 1, 2, and 3 and subsequently calculated the accuracy. This highlights the dataset's out-of-distribution (OOD) nature, showing the signal differences caused by different postures. From Figure 9, it can be seen that the accuracy of the same gestures significantly decreases in Posture 2 and Posture 3, further demonstrating the OOD problem.

About future suggestions, the extension of SSFDA to other distribution shift scenarios. We appreciate this excellent suggestion for our future work. You are correct that our current SSFDA method primarily focuses on distribution shifts caused by forearm posture variations. We acknowledge that this is an important area for expansion. In the revised version, we will extend our discussion in the future work section to address this point. We plan to outline our strategy for adapting SSFDA to handle other sources of distribution shift, such as electrode displacement, changes in skin conditions, and variations in muscle fatigue. This expansion will involve modifying our membrane potential memory mechanism and pseudo-label generation process to account for these additional factors.

To reviewer: Thank you for your insightful review of our paper. We are thrilled that you found the introduction of Jaccard Attention to be an innovative and computationally friendly algorithm for SNNs. Your recognition of our pioneering work on source-free domain adaptation in SNNs and its enhancement of domain generalization performance is highly appreciated. We are also glad that you valued our comprehensive validation across multiple datasets and the open-sourcing of our code for reproducibility. Your constructive feedback and suggestions will significantly guide our future research. We sincerely appreciate your valuable time and effort in reviewing our work.

About how the probability $P$ is determined when generating pseudo-labels. Thank you for your question. The probability $P$ is determined by selecting the mode (most frequent value) among the top k membrane potentials as the high-probability pseudo-label. The probability $1-P$ involves randomly selecting a pseudo-label from the top k membrane potentials. The value of $P$ ranges between $0$ and $1$. In our validation, we searched the space from $0.1$ to $0.9$ in increments of $0.1$ to determine the optimal value.

About extending Jaccard Attention to 3D data. Thank you for your insightful question regarding extending Jaccard Attention to 3D data. To address this, we have extended the original Jaccard Attention formula to handle 3-dimensional data. The original equation is:

$

\mathrm{SJA}\left(\mathbf{Q}, \mathbf{K}\right) = \frac{\sum_{ij}\min\left(q_{ij}, k_{ij}\right)}{\sum_{ij}\max\left(q_{ij}, k_{ij}\right) + \epsilon} \ \mathbf{V}

$

For 3D data, we extend the indices $(i,j)$ to $(i,j,k)$ to account for the additional dimension. The extended equation is:

$

\mathrm{SJA}\left(\mathbf{Q}, \mathbf{K}\right) = \frac{\sum_{ijk}\min\left(q_{ijk}, k_{ijk}\right)}{\sum_{ijk}\max\left(q_{ijk}, k_{ijk}\right) + \epsilon} \ \mathbf{V}

$

Here, $q_{ijk}$ and $k_{ijk}$ represent the elements of the 3D tensors $\mathbf{Q}$ and $\mathbf{K}$ respectively. The summations now run over all three dimensions $i$, $j$, and $k$. This extension allows us to apply the Jaccard Attention mechanism to 3-dimensional data, maintaining its computational efficiency and enhancing feature representation in SNNs.

About demonstration of the domain shift between different postures. The characteristics of the sEMG distribution shifts corresponding to different forearm postures are demonstrated in Appendix Figure 9. We used data from Posture 1 for inference on data from Postures 1, 2, and 3 and subsequently calculated the accuracy. This highlights the dataset's out-of-distribution (OOD) nature, showing the signal differences caused by different postures. Figure 9 shows that the accuracy of the same gestures significantly decreases in Posture 2 and Posture 3, further demonstrating the OOD problem.