Leveraging characteristics of the output distribution for identifying adversarial audio examples

Many thanks for your valuable feedback. We performed further investigations based on your comments and made updates to our paper accordingly.

Q: Detection performance is only evaluated on limited adversarial attack methods. The authors only evaluate their method on C&W attack and Psychoacoustic attack. More attack methods like [1] [2] and even some black box methods like FAKEBOB [3] are still needed to be included to prove the general performance of the detection method.

A: Thanks for your suggestions, we extended our analysis to three more attacks: PDG and those proposed in [1] and [2]. We evaluated the detection performance of our classifiers across diverse attacks. Our findings reveal that our GCs utilizing mean-median characteristic and NNs exhibited successful transferability to the Psychoacoustic attack. Moreover, when exposed to untargeted attacks, they outperformed the baseline in comparison to PGD and Kenansville. For enhanced clarity, we introduced a new analysis, presented in the results section 5.3 on pages 7-8 and table 6 of our revised manuscript.

We did not include FAKEBOB in our analysis, since it is designed specifically for speaker recognition systems, which falls outside the scope of our research.


Q: Lack of comparison with other audio adversarial example detection methods like [4].

A: Thanks for your input. We included [4] as a baseline in our analysis, but found that it is less effective than our method. Please refer to section 5.3, page 8, and table 4 in our latest manuscript.


Q: In Section5.1, four guiding principles are provided on selection process. Do such principles limit the adversarial attack implementation? Would there be other attack scenarios like tokens insertion or deletion?

A: Thank you for expressing your concern. The decision to limit the audio length to up to five seconds, was a strategy compromise, balancing time and resources, as generating longer adversarial examples take much more time.

The decision to choose unique target transcriptions is primarily made to minimize bias in our results, ensuring the inclusion of the broadest possible variety of selected words.

The choice of the number of tokens for constructing a target transcription is influenced by the nature of the duration of an audio. There is always a limit of the number of tokens we can hide in an adversarial example, for shorter inputs, achieving a longer target transcription becomes more challenging, and vice versa. Even when the total number of tokens in both the original and target transcriptions are equal, the transcription length itself may vary, as shown in our Demo: https://confunknown.github.io/characteristics_demo_AEs/

This design choice is also supported by previous research. For example, [5] conducted an evaluation on the rate of phones that could be inserted per second without compromising adversarial attack performance. Their findings indicated that as the phone rate increases, the WER also increases. They observed four phonemes per second as a reasonable choice.

[5] Schönherr et al., Adversarial attacks against automatic speech recognition systems via psychoacoustic hiding. In Network and Distributed System Security Symposium (NDSS), 2019.

Many thanks for your feedback! We run additional experiments with the suggested attacks and updated our paper accordingly.

Q: Why not consider other attack methods, such as PGD attacks or Transaudio transfer attacks?

A: We expanded our empirical analysis by incorporating additional experiments that explore three untargeted attacks: PGD, Kenansville and Genetic attacks. While we made an effort to incorporate the Transaudio attack, constraints on time prevented us from creating an implementation specifically designed for Speechbrain and discussing certain inquiries with the authors. To overcome this, we opted to assess our detectors against two additional black-box attacks—Kenansville and the Genetic—suggested by reviewer NsBd. We evaluated the detection performance of our classifiers across diverse attacks. Our findings revealed that our GCs utilizing mean-median characteristic and NNs exhibited successful transferability to the Psychoacoustic attack. Moreover, when exposed to untargeted attacks, they outperformed the baseline in detecting PGD and Kenansville attacks. For enhanced clarity, we introduced a new analysis, presented in the results section 5.3 on pages 7-8, and table 6 of our revised manuscript.

We thank for your valuable feedback, we have taken your suggestions into consideration and made updates to our paper accordingly.

Q: Have you looked into ways to make the classifiers more robust?

A: During the training of all models, to improve generalization, we applied standard data augmentation techniques provided in SpeechBrain: corruption with random samples from a noise collection, removing portions of the audio, dropping frequency bands, and resampling the audio signal at a slightly different rate. We described in the paragraph “ASR system” that is in Section 5.1, Page 5.


Q: Have you evaluated the computational overhead added by extracting the distributional features and running the classifiers? Is this method efficient enough for real-time usage in production systems?

A: Thanks for asking, we performed experiments to measure the total time the system takes to predict 100 audio clips, utilizing an NVIDIA A40 with a memory capacity of 48 GB. As a result, running the assessment with our detectors took approximately an extra 18.74 ms per sample, therefore the proposed method is suitable for real-time usage. We have incorporated this information into the paper, and provided additional details in Appendix A.3 on page 16.


Q: Can the optimization problem for adversarial attacks be framed to minimize divergence from the benign data distribution while still deceiving the ASR system? And what challenges exist in developing adaptive attacks that maintain summary statistics while effectively fooling the system?

A: Thanks for the suggestion. This is an interesting idea. Unfortunately, due to the limited amount of time, we did not manage to finalize experiments with the suggested adaptive attack, but we will continue investigating and add the results to the final paper.

We value your insightful comments and queries. To address your concerns, we provide explanations in the following and updated the paper accordingly.

Q: Why are these statistical features selected?

A: Previous work indicated the high potential of statistics of the output distribution that implement uncertainty measures: on the one hand, the mean entropy was turned into a defense method against adversarial attacks on hybrid ARS system with limited vocabulary, as discussed in [1]. On the other hand, the Kullback-Leibler divergence between the distributions in consecutive steps was used to assess the reliability of ARS systems, see [2]. We therefore aimed to investigate if this also applies for state-of-the-art end-to-end ARS systems. We expanded our analysis to simple characteristics (like max, min, median) and were surprised ourselves that they lead to sometimes even better detection results.

[1] Däubener et al., Detecting adversarial examples for speech recognition via uncertainty quantification. In Proc. Interspeech 2020, pp.4661–4665, 2020.

[2] Meyer et al., Performance monitoring for automatic speech recognition in noisy multi-channel environments. In 2016 IEEE Spoken Language Technology Workshop (SLT), pp. 50–56, 2016.


Q: Why the generated adaptive adversarial audio examples are noisier when optimizing with respect to relevant feature?

A: The adaptive attack contains two (potentially competing) loss functions, which makes the optimization problem harder. We also were successful in generating less noisy examples with the adaptive attack (see Appendix A.1, page 14) which however were less often fooling the detection model. We added a sentence to make this clearer in the main paper.


Q: Will be the detection model that is trained on one specific kind of adversarial example be generalizable to other types of adversarial examples?

A: Yes. First, note that the construction of Gaussian classifiers can be done only with benign examples. While we also investigated the performance based on the best characteristic picked on a validation set, we also found that some characteristics perform very well for different attack types and thus can be chosen without any adversarial data. Only benign data for picking the threshold is needed. Moreover, we conducted experiments on intra-model generalization for the neural networks, investigating if a detection model trained on the C&W attack can be applied as a defense method against the other targeted (i.e., Psychoacoustic) and three newly added untargeted attacks. We found that our GCs based on the mean-median characteristic and NNs demonstrated effective transferability to the Psychoacoustic attack. Additionally, when subjected to untargeted attacks, they performed better than the baseline against PGD and Kenansville attacks. To make this more clear, we incorporated a new analysis displayed by the results in section 5.3/pages 7-8 /Table 6 of our updated manuscript.


Q: Will a detector trained on one ASR model be able to detect adversarial examples that are generated on a different ASR model?

A: We don't expect our detectors to generalize between models, since the statistics for different models differ. Previous research has demonstrated almost no transferability of adversarial examples between speech models, as indicated in [1]. This lack of transferability, especially in the context of targeted optimization attacks, suggests that adversarial examples crafted to target a specific model lose their efficacy when applied to a new ASR model, posing a limited threat. We verified this claim with our models and obtained similar results, with almost no transferability observed between models. These results are reported in Appendix A.11 on pages 30-31.

[1] Abdullah et al., Sok: The faults in our ASRs: An overview of attacks against automatic speech recognition and speaker identification systems. 2021 IEEE Symposium on Security and Privacy (SP), pp. 730–747, 2020.


Q: About the adaptive attack, have the authors considered other types of attacks that may decrease the noise level of the adversarial audio examples?

A: Yes, we did. As described in details in Appendix A.1 (page 14) we experimented with several different settings that resulted in lower noise levels but also in less affective attacks. We added a sentence to make this clear in the main paper.

Thank the authors for the detailed clarifications. I appreciate it. However, I will still maintain my original rating due to the limited effectiveness of the proposed approach against adaptive defenses and the limited explanation of the working mechanism behind the approach. My suggestions would be to explain why the selected features work effectively, especially compared with the end-to-end detector, or show the potential of the proposed method against adaptive attacks under a certain bound.

Including an additional empirical analysis to the list:

5- the detectability of an adaptive attack with slightly different adversarial loss function, incorporating a penalty to minimize the statistical distance from the benign data distribution (as recommended by jXR8), see Appendix A.1 and Table 12 on page 14 of our revised manuscript.