Training Adversarially Robust SNNs with Gradient Sparsity Regularization

Thanks for the valuable comments. We are encouraged that you find the tackled problem is relevant to the community, our work original and our experimental results good. We would like to address your concerns and your questions in the following.

Q1: Discuss more in detail the key findings and limitations in the existing literature, and how the challenges are solved in this paper.

Thanks for your valuable feedback! The key findings and limitations in the existing literature are as follows. Methods for improving the robustness of SNNs can be broadly categorized into two classes. The first class draws inspiration from ANNs. A typical representative is adversarial training and its variants. This approach has been shown to effectively defend against attacks that are used in the training phase. Another method is certified training, whose application to SNNs remains challenging. Current efforts in this area have primarily focused on the MNIST dataset. The second category consists of SNN-specific techniques designed to enhance robustness, such as using the Poisson encoder. However, the Poisson encoder generally yields worse accuracy on clean images than the direct encoding, and the robustness improvement caused by the Poisson encoder varies with the number of time-steps used.

In this paper, we propose the SR strategy which can be combined with adversarial training to boost the robustness of SNNs. The SR strategy is supported by a solid theoretical foundation, and experimental results on CIFAR-10 and CIFAR-100 demostrate its effectiveness in enhancing the robustness of SNNs.

We have revised Sec. 2.2 to provide a more detailed discussion of the key findings and limitations in the existing literature in terms of SNN robustness, as well as how the challenges are solved in this paper.

Q2. The experiment settings used to generate Figure 1.

The experimental settings in Figure 1 are outlined as follows. The SNN architecture is a VGG-11 model. The adversarial attack employed is FGSM, with $\epsilon$ ranging from $0.00$ to $0.25$. For the random attack on SNNs, a perturbation $\delta$ is randomly generated to satisfy a uniform distribution within the hypercube $\Vert \delta \Vert_\infty \leqslant \epsilon$. Subsequently, $\delta$ is added to the input image $x$ and the pixel value of the perturbed image are clipped to the range of [0, 1]. The classification accuracy is evaluated for all $x+\delta$, where $x$ is drawn from the test set of CIFAR-10 (Figure 1(a)) and CIFAR-100 (Figure 1(b)). We have added these detailed settings in Sec. 4.1 of the revised version.

Q3: Discuss more clearly the differences between the approximation of the gradient regularization term employed in this paper and the related works.

Regarding the approximation method for computing the gradient norm, both our paper and the related work (Finlay & Oberman 2021) use the finite differences technique. The only difference is that, we take the sign of the gradient as the difference direction while Finlay & Oberman took the sign direction divided by $\sqrt{N}$, where $N$ is the dimension of $x$. For other differences between the objectives of regularization terms in this paper and the related work, please refer to "To All Reviewers".

Q4: Discuss in detail all hyperparameters used to conduct the experiments, and provide the code.

We would like to clarify that the setting of hyperparameters are discussed in Appendix C and D. Thanks for your suggestion! We have already uploaded the code in the revised supplementary material.

Q5: Please Discuss the limitations and potential solutions to overcome accuracy loss for clean inputs casued by SR.

Thanks for the valuable suggestion! The enhancement of adversarial robustness through SR is accompanied by a notable loss in accurac on clean images. In future work, we aim to achieve a better trade-off between classification accuracy and adversarial robustness. One possible avenue we plan to explore is the utilization of the simulated annealing algorithm in SR* to dynamically adjust the weight assigned to the regularization term, potentially leading to a more balanced outcome in terms of both accuracy and robustness.

We have added these limitations and potential solutions in Section 6 of the revised paper.

Dear reviewer FfwP, thanks again for your careful and valuable comments! Since the rebuttal discussion is due soon, we’ll be appreciated to know whether our replies have addressed your questions. If there are any further clarifications required or any other concerns, please feel free to contact us. Many thanks!

Thank you for your valuable insights. We sincerely appreciate your advice regarding our work. We apologize for any ambiguities in our description of the SR method that may have caused misunderstandings. In this rebuttal, we would like to first clarify the distinction between SR and adversarial training. Subsequently, we will address your concerns and provide responses to your questions. If you have any further suggestions or questions, please feel free to let us know.

Q1: Clarification of the SR strategy

In Theorem 1 of the main text, we have proved that the disparity between the adversarial vulnerability and random vulnerability is upper bounded by the sparsity of $\nabla_x f_y$. The main idea of our SR strategy is to regularize the value of $\Vert \nabla_x f_y(x,w) \Vert_0$. It is important to note that in our approach, $f(\cdot, w)$ represents the model function, which differs from previous methods that utilize the multi-class calibrated loss function as a regularization term. Here, the multi-class calibrated loss function $l(f(x,w),y)$, such as cross-entropy, is the loss function used to train a classifier where $y$ is the true label of $x$. In other words, our proposed method introduces a regularization term that remains independent of the choice of loss function.

To achieve the regularization of $\Vert \nabla_x f_y(x,w) \Vert_0$, we use $\Vert \nabla_x f_y(x,w) \Vert_1$ to approximate the $\ell_0$ norm of the gradient and then use finite difference method to make the regularization of $\Vert \nabla_x f_y(x,w) \Vert_1$ computationally feasible. Theoretically, if direct backpropagation on $\Vert \nabla_x f_y(x,w) \Vert_1$ were feasible, finite differences could be bypassed. However, as reported in Finlay & Oberman's work, directly calculating the backpropagation of $\Vert \nabla_x f_y(x,w) \Vert_1$ on ANNs is very time-consuming. Additionally, we found that SNNs are not trainable with such double backpropagation approach.

We also find that the demonstration of Algorithm 1 in the main text is not clear enough, potentially leading readers to confuse our proposed SR algorithm with the adversarial training method and its variants. We would like to clarify that, as depicted in Algorithm 1 (Line 5 to Line 7), $\hat{x}^i=x^i + hd^i$ is fundamentally different from an adversarial example in terms of both numerical value and meanings. Actually, it is used to calculate the difference quotient

$

\left| \frac{ (f_{y^i}(\hat{x}^i, w)) - (f_{y^i}(x^i, w)) }{ h } \right| \tag{3-1}

$

which is the approximation of $\Vert \nabla_{x^i} (f_{y^i}(x^i, w)) \Vert_1$. While theoretically, $d^i$ can be arbitrary, it is set to be the sign of the gradient in this paper. This choice aligns with the objective of approximating the $\ell_1$ norm of the gradient, as explained in detail in Appendix B.

Additionally, $h$ represents the step size used for the approximation, rather than the attack strength. It is recommended to choose a small value for $h$ in Equation (3-1) to achieve a more accurate estimation of the $\ell_1$ norm. As $h$ increases, the approximation results become less accurate, leading to a decrease in robustness. However, if $h$ were the attack strength in one-step FGSM, larger $h$ would contribute to better robustness (at the expense of decreased accuracy on clean images). The impact of $h$ in our training algorithm is different from the impact of $h$ as the attack strength in FGSM adversarial training. In summary, the SR algorithm does not involve the calculation of adversarial samples, making it fundamentally different from adversarial training and its variations.

We have revised the parts of the manuscript that may cause misunderstandings and added this clarification in Appendix G.

Q2: Differences between our training algorithm and the work by Finlay & Oberman 2019.

Please refer to "To All Reviewers".

Q3: Differences between SR and (clean) logit pairing approach. Differences between SR and (adversarial) logit pairing approach, as well as TRADES.

Please refer to "To All Reviewers".

Q7: Following my question above, Table 2 is a bit complicated in how SR and AT can be disentangled. In the case of ablation models without AT but SR, how were the adv. examples to compute the regularization term obtained?

As clarified in the "Clarification of the SR strategy" section, the finite difference approximation employed in SR is not a type of adversarial training using one-step FGSM. Thus, SR is not entangled with AT, and the ablation models without AT but with SR do not involve adversarial examples. To provide further clarity on the distinction between SR and AT, we present the training loss function used in different training algorithms.

For PGD5-AT: $

\mathcal{L}(x^i, y^i,w) = \text{CE}(f(x_{adv}^i), y^i)
\tag{3-3}

$

For single SR: $

\mathcal{L}(x^i, y^i,w) = \text{CE}(f(x^i), y^i) + 
    \lambda_{\text{SR}}\left\Vert \nabla_{x^i} f(x^i,w)\right \Vert_1
    \tag{3-4}

$

For SR*(SR+PGD5): $

\mathcal{L}(x^i, y^i,w) = \text{CE}(f(x_{adv}^i), y^i) + 
    \lambda_{\text{SR}}\left\Vert \nabla_{x_{adv}^i} f(x_{adv}^i,w)\right \Vert_1
    \tag{3-5}

$

Q8: If assumptions in this paper would also hold for SNNs that use Poisson input coding, or SNNs that use LIF?

Thanks for pointing it out. Indeed, the SR strategy can be applied theoretically as long as SNN is differentiable through methods such as surrogate gradients.

To demonstrate this point, we apply SR to SNNs with VGG-11 structure consisted of LIF neurons on the CIFAR-10 dataset. The experimental results shown in Table R5 clearly demonstrate the effectiveness of the SR* strategy in enhancing the robustness of SNNs with LIF neurons. Models trained with SR* exhibit significantly improved robustness against all PGD attacks compared to models trained with AT, with a 4% decline in classification accuracy on clean images.

Table R5-1. Experiments on SNNs with LIF neurons

We also trained two VGG-11 models on CIFAR-10 using Poisson encoding. During training, we use the same setting as reported in the main text with the only different being the input encoding. When evaluating robust performance, the Poisson input encoding can be treated as an input transformation and use the EOT method (Athalye, Anish, Nicholas Carlini, & David Wagner) to attack. According to the results in Table R5-2, the performance of SR model is higher than the Vanilla model and the robust performance of SNN trained with SR is even 15% higher than the vanilla SNN under PGD10 attack. That proves the effectiveness of the sparsity regularization.

Table R5-2. Experiments on SNNs using Poisson Encoding

These experiments demostrate that our strategy also holds for SNNs with LIF neurons or with Poisson encoding.

Q9: Experiments on DVS datasets

We use the VGG-11 architecture on DVS-CIFAR10 Dataset and compared Vanilla trained model and SR trained model. When evaluating, we perform the adversarial attack on preprocessed frames and add adversarial noise onto each frame. The robustness performance is shown in Table R6. The performance of SR trained model is significantly higher than the Vanilla model, which proves the effectiveness of SR method.

Table R6. Performance of models on DVS-CIFAR10

Q10: The computational overhead of SR and SR*, in comparison to simple AT or RAT with PGD-5 as well.

The computational costs for one epoch training of various algorithms, including vanilla, PGD5 AT, RAT, SR, and SR*(PGD5+SR), on CIFAR-10 using the VGG11 architecture are summarized in Table R6. From the table, we observe that single SR incurs a computational cost 1.5 times that of RAT but consumes less than half the time needed by PGD5 AT. The computational cost of SR* is the highest among all training algorithms since it combines both SR and AT. However, models trained with SR* acheive the best robustness compared to models trained with other methods. We have added this comparison in Appendix I.

Table R7. The computational cost in one epoch of different training algorithms

Q4: Further clarification of an "ensemble attack" for white-box evaluations.

We would like to clarify that as dicussed in the Appendix D, to avoid gradient obfuscation, the ensemble attack involves utilizing a diverse set of attacks with different surrogate gradients and gradient backward methods. For each test image, we applied various attacks with the following setting. We consider an ensemble attack to be successful for a test sample as long as the model is fooled by any of the attacks from the ensemble.

Table R3. The experimental settings for the ensemble attack

Q5: Extensive evaluation with for instance $\gamma \in [0.1,3.0]$.

Thanks for your valuable suggestion, we have applied the attack with $\gamma \in [0.1, 3.0]$ and Table R4 reported the results of a PGD10 attack on VGG-11 models with different training algorithms on the CIFAR-10 dataset.

We compare three different attack combinations to evaluate the impact of different surrogate functions on the attack strength. We selecte RAT, PGD5-AT and SR*(PGD5+SR) model as the target models. For PGD10(w/o ensemble), we only use the Triangle-shaped surrogate function， which is identical to the one used in training. For PGD10 (w/ ensemble), we use the attack combination as described in the manuscript. For PGD10($\gamma$ from [0.1, 3.0]), we incorporate 30 different Triangle-shaped surrogate function with its $\gamma$ ranging from [0.1, 30], as suggested.

We find that both ensemble attack methods significantly improve attack performance and mitigate the impact of gradient obfuscation. This indicates that both shape and width of the surrogate function can influence the capability of the adversary. In addition, we acknowledge that the PGD10($\gamma$ from [0.1, 3.0]) attack is slightly more effective than the ensemble attack used in this paper. However, it is important to note that the PGD10($\gamma$ from [0.1, 3.0]) attack uses a 30-fold fine-grained grid search over attack hyperparameters for each image, which is considerably more computationally expensive compared to the ensemble attack used in our paper.

In conclusion, on one hand, we agree with reviewer HCoc's claim that both width and shape of the surrogate function can impact attack performance. We have included additional discussions about this in the limitation part. On the other hand, we would like to clarify that our contribution mainly lines in the defense algorithm, not the attack algorithm. In our paper, we use the same attack method for all comparisons to ensure a fair comparison of different methods. We believe that whether we use our current method or incorporate the attack combinations suggested by reviewer HCoc, it will not significantly influence the effectiveness of our defense strategy.

Table R4. The classification accuracy (%) under the ensemble attack with different γ

Q6: Models were trained using PGD-5 adversarial examples, but the Algorithm 1 denotes adv. examples obtained via one-step FGSM. Which one is correct?

Thanks for pointing it out! We would like to clarify that in the SR model, we do not use adversarial examples obtained via one-step FGSM to conduct adversarial training, as explained in "Clarification of the SR strategy". For SR* models, we employ PGD-5 as the general adversarial training approach, and the training loss function is

$

\mathcal{L}(x^i, y^i,w) = \text{CE}(f(x_{adv}^i), y^i) + 
    \lambda_{\text{SR}}\left\Vert \nabla_{x_{adv}^i} f(x_{adv}^i,w)\right \Vert_1
\tag{3-2}

$

where $\left\Vert \nabla_{x^i} f(x^i,w)\right \Vert_1$ is approximated by the finite differences, and $x_{adv}^i$ is obtained by PGD-5.

Q11: Include adversarially trained models in Figure 3.

We have added the gradient distribution of AT-trained models with respect to the input image in Figure 3. The distribution of gradient values for SNNs-SR* is more concentrated around zero compared to that of vanilla SNNs and SNNs-AT.

Q12: Minor comments

Thanks for pointing it out. We correct the "$\ell_p$ norm" at the end of Sec 3.2 in the revised version.

Dear reviewer HCoc, we sincerely hope our posted response can help to address your concerns on our paper and serve as a reference for your re-assessment of our work. If you have any further comments and questions, please let us know and we are glad to write a follow-up response.

Thanks for your valuable comments and suggestions. We are delight that you find our obervation meaningful, our idea interesting and experiments detailed. We would like to address your concerns and your questions in the following.

Q1: Details of Formula 10

Thanks for pointing it out. We would like to clarify the notations used in Formula (10) as follows. Let $x$ denote the image, and $\{x[1], x[2], \dots, x[T]\}$ represent the input image series. In our paper, we use $x[t]=x$ for all $t=1,\dots,T.$ The network is denoted by $f$ with parameters $w$. The output of the network $f$ is a vector $f(x,w)\in \mathbb{R}^{N \times 1}$, where $N$ represents the number of classes. The output vector $f(x,w)$ is obtained by applying the softmax function to the output of the last layer, i.e.

$

f(x, w) = softmax \left( 
    \sum_{t=1}^T s_1^L[t], \dots, \sum_{t=1}^T s_N^L[t]
\right).
\tag{2-1}

$

Therefore, the $y^{th}$ component of $f(x, w)$ where $y$ represents the true label is

$

f_y(x, w) = softmax_y \left( 
    \sum_{t=1}^T s_1^L[t], \dots, \sum_{t=1}^T s_N^L[t]
\right).
\tag{2-2}

$

According to the chain rule, the gradient of $f_y$ with respect to the input $x$ is

$

\begin{split}
    \nabla_x f_y(x,w) &=  \frac{\partial softmax_y }{\partial \sum_t s_1^L[t]}
                 \cdot
                \nabla_x \left( \sum_t s_1^L[t] \right) + \dots  + 
                \frac{\partial softmax_y }{\partial \sum_t s_N^L[t]}
                 \cdot \nabla_x \left( \sum_t s_N^L[t] \right).
\end{split}
\tag{2-3}

$

In Equation (2-3), the gradient of the $i^{th}$ component $\nabla_x \left( \sum_t s_i^L[t] \right)$ in the last layer with respect to $x$ can be further expresses as

$

\nabla_x \left( \sum_t s_i^L[t] \right) =  \sum_t \nabla_x s_i^L[t] 
= \sum_t \sum_{\tilde{t}=1}^t \nabla_{x[\tilde{t}]} s_i^L[t] .
\tag{2-4}

$

Finally, the gradient $\nabla_x f_y(x,w)$ is written as

$

\nabla_x f_y(x,w) = \sum_{i=1}^N \left( \frac{\partial softmax_y
                }{
                    \partial \sum_t s_i^L[t]
                }
                \left(
                    \sum_{t=1}^T \sum_{\tilde{t}=1}^t \nabla_{x[\tilde{t}]} s_i^L[t]
                \right) \right).
\tag{2-5}

$

We revise Formula (10) in the latest version, and provide the detailed derivation of the formula in Appendix J.

Q2: Whether ANN can adopt SR?

Thanks for pointing it out! Theorem 1 also holds true for ANNs which are differentiable. Therefore, this technique can be employed to enhance the robustness of ANNs. To demonstrate this point, we conduct experiments for ANNs with VGG11 architecture on the CIFAR-10 dataset. The classification accuracy (%) is reported in Table R2-1, which demonstrates that our can also boost the robustness of ANNs.

Table R2-1. Performance of ANNs on CIFAR-10 dataset

Q3: Whether the proposed method will affect the robustness under random attack?

Thanks for your suggestion. To evaluate that, we report the classification accuracy (%) of models trained with SR and SR*(PGD5+SR) under random attack (eps=0.1) with WideResNet-16 on CIFAR datasets. The results are shown in Table R2-2, we can find that the SR and SR* strategy can also improve the robustness of SNNs against random attacks.

Table R2-2. Compare the random robustness of different methods on CIFAR datasets

Thank you for your thoughtful comments. We are encouraged that you find our paper technically sound, well-written and our results significant. We would like to address your concerns and your questions in the following.

Q1: Experiments on more large-scale datasets

We trained three VGG-11 models on the TinyImageNet dataset: a vanilla model with no robustness enhancement (Vanilla), an FGSM [eps=2/255] adversarially trained model (AT), and a model using both FGSM [eps=2/255] adversarial training and sparsity regularization (SR*). Throughout training and evaluation, we cropped all images to a resolution of 64x64, which is more complex compared to the 32x32 images from CIFAR-10 datasets.

Under these conditions, the SR* model exhibited the best performance, achieving an accuracy of 10.82% under PGD50 attack and 14.89% accuracy under FGSM attack. In comparison to the pure AT model, the SR* model showed significantly better robustness against various attacks, highlighting the effectiveness of our method.

Table R1. Performance on TinyImageNet dataset

Q2: Add assumptions to Theorem 1

Thanks for your suggestion! In our revised version, we have incorporated the assumptions that the function $f$ is differentiable and that $\epsilon$ is sufficiently small into Theorem 1.

Q3. Other minor questions.

Thanks for pointing it out! We have revised these typos and updated our paper.

The differences between our work and (adversarial) logit pairing approach (Kannan, Kurakin & Goodfellow, 2018), as well as TRADES (Zhang et al. ICML 2019).

• Both adversarial logit pairing approach (RFGSM-40) and TRADES (solving a maximum problem) utilize adversarial examples during their training phase. However, it is important to clarify that the proposed SR strategy does not use adversarial examples as explained in the first response to Reviewer HCoc. In other words, SR is not a kind of adversarial training.
• The objectives are different. The primary objective of the adversarial logit pairing and TRADES is to encourage the probability distributions of the input image and its corresponding adversarial example to be similar. In contrast, our proposed SR strategy aims to induce sparsity in the gradient $\nabla_x f_y(x,w)$. We are not targeting the regularization of the probability distribution, and the regularization term corresponds to the gradient of $f_y(x,w)$, rather than the output vector $f(x,w)$. From an intuitive perspective, minimizing the $\ell_0$ norm of $\nabla_x f_y(x)$ serves to bring $x$ closer to a local maximum point, making it harder for attackers to generate adversarial examples through gradient-based methods.