DeSparsify: Adversarial Attack Against Token Sparsification Mechanisms

Thank you for your time, effort and comments.

Q1: "..limited to vision transformers.."

A1: Our study focuses on vision transformers because the token sparsification methods we examine are specifically designed and optimized for this domain. We agree that exploring the applicability of DeSparsify to other domains such as NLP and speech recognition is important for demonstrating its broader relevance. However, extending the TS methods to these domains, first requires significant research effort for adapting them to these models, which were beyond the scope of the current study. We recognize the potential of DeSparsify in other domains and intend to pursue this direction in future work. By first establishing a solid foundation in the vision transformer domain, we aim to build a robust framework that can later be adapted and applied to a wider range of models and tasks.

Q2: "..countermeasures are not compared.."

A2: In our study, we have presented a comprehensive analysis of the proposed countermeasures, including two different token selection strategies, and examined their effects in terms of both availability (GFLOPS) and integrity (accuracy) on benign and adversarial samples, as detailed in Appendix B. Our results demonstrate that the proposed countermeasures successfully mitigate the impact of adversarial samples on the model's availability, reducing GFLOPS to levels close to those observed with clean images. Additionally, the countermeasures maintain nearly similar accuracy on clean images, indicating that they do not interfere with the model's performance on benign samples. Furthermore, to the best of our knowledge, no existing adversarial defense mechanisms specifically address availability-based attacks. Therefore, our work highlights a novel and important aspect of adversarial defenses that has not been previously explored.

Q3: "..techniques such as SlowFormer.."

A3: We thank the reviewer for pointing out the work on SlowFormer [1]. We were not aware of this work since it was published in conference proceedings on June 15th, while our work was submitted a month earlier on May 15th. We believe that the publication of such work in a top-tier conference emphasizes the relevance and importance of exploring the vulnerabilities of the TS domain. However, upon reviewing the mentioned work, we believe our research introduces novel contributions to the field, as well as offers more extensive and detailed insights. Specifically, in our study:

a) we propose various attack variants, including: single, class-universal, universal, and universal patch, as opposed to SlowFormer that only evaluated a universal patch;

b) we propose novel loss functions that stem from a deep understanding of each TS technique, as opposed to SlowFormer that simply uses the TS technique loss function.

c) we use three different model architectures including DeiT-t, DeiT-s, and T2T-ViT. Some of these models were not published by the authors, which we trained ourselves. In SlowFormer, they only used the provided models.

d) Our experiments dive deeper into each TS technique, providing novel insights and intuitions. We present evaluations on TS mechanism transferability and ensemble strategy, and show the practical effect of the attack on GPU hardware.

e) we propose tailored countermeasures and present a comprehensive analysis on the trade-offs between availability and integrity. In addition, our countermeasure can be plugged into any TS technique without requiring model finetuning, in contrast to SlowFormer that only proposes a naive adversarial training strategy which is not clearly described.

Q4: "..transferability analysis.." + "..assumption that the adversary has access.."

A4: Please refer to the general comment for further details.

Q5: "..more concrete examples.."

A5:

We discuss potential real-world scenarios where our attack on token sparsification (TS) techniques could be applied, highlighting the impact and applicability of the attack.

Surveillance cameras scenario: consider a cloud-based IoT platform that uses vision transformers with TS techniques to process and analyze images from a network of surveillance cameras that monitor various locations and send data to a centralized cloud server for real-time analysis and anomaly detection.

Attack impact: increasing computational overhead and latency could lead to delays in detecting anomalies, potentially allowing security breaches to go unnoticed for longer periods. In a high-security environment, such a delay could have severe consequences, compromising the safety and security of the monitored locations.

Autonomous drones scenario: consider autonomous drones that navigate and analyze the environment using models with TS techniques. For example, drones that are used for delivery services, agriculture, and surveillance.

Attack impact: An adversarial attack could overload the drone’s computational resources (leading to rapid battery depletion and overheating) that cause navigation errors, reduced flight time, or complete system failure. These can result operational inefficiencies or accidents, especially in complex environments where precise navigation is crucial. In critical applications, such an attack could incapacitate the device, leading to mission (e.g., rescue) failure or safety hazards.

Wearable health monitors scenario: consider wearable health monitors that analyze physiological data, such as heart rate, activity levels, and sleep patterns. These devices provide real-time health insights and alerts to users.

Attack impact: an attack could lead to incorrect health metrics and delayed alerts. This could affect the user's health management, potentially missing critical health events that require immediate attention.

Q6: "The code.."

A6: We have enhanced the documentation of the code, remove hard-coded paths, provided setup scripts, and created a reproducability guide.

Thank you for your time, effort and comments.

Q1: "The submission is mainly held back by the writing quality..."

A1: Thank you for your detailed comments. We fixed the presentation issues and corrected them in the paper.

Q2: "The main technical drawback of the attack.."

A2: Please refer to the general comment. We have included a detailed analysis of different attacker knowledge scenarios.

Q3: "The paper's threat model is mainly focused on the classification task.."

A3: The focus on the classification task mainly stems from the fact that the dynamic token sparsification methods we examine in our paper have only addressed this task. Applying these methods to other domains, such as generative and image-to-text tasks, first requires significant research effort for adapting them to these models, which is beyond the scope of our current study. Nonetheless, this represents a potential area for future work, as we recognize the importance of extending our findings to the broader applications of vision transformers. It is also worth noting that since token sparsification is a novel domain, it is common practice for adversarial attacks to initially focus on image classification before being extended to other domains in subsequent research.

Q4: "In Section 4.2.1 it is mentioned that the token sampling function..."

A4: In other words, the significance scores are used to calculate the mapping function between the indices of the original tokens and the sampled tokens. To determine which tokens remain active in the current block, ATS uses a fixed sampling scheme $k = \{ \frac{1}{2K}, \frac{3}{2K}, \ldots, \frac{2K-1}{2K} \}$ that is equally distributed in the range [0,1], where $K$ is chosen to be the maximum number of tokens in block 0 (for DeiT-s, $K = 197$, as there are 196 tokens plus an additional class token). For instance, if the first token has a high significance score (e.g., 0.1), the inverse of the CDF will return the index of the first token for all $k$ values such that $k < 0.1$. In the example above, the first 20 entries of $k$, i.e., $\{ \frac{1}{394}, \ldots, \frac{39}{394} \}$, will map to the index of the first token. Since ATS only keeps one instance if a token is sampled more than once, the number of active tokens in the current block dramatically decreases. This will be clarified in the final version.

Thank you for your time, effort and comments.

Q1: ".. This objective conflicts with the operations described in Eqs. (2-3)."

A1: Please note that the novel loss attacking components we propose should be minimized and not maximized. That is, our optimization process follows the computed gradient direction, as opposed to the classic PGD attack which goes to the opposite direction. Indeed, in the implementation of the PGD baseline (denoted in the paper as Standard PGD), we negate the cross-entropy loss term to perform the attack. Therefore, there is no conflict between the operations, which is also evident in the ablation study results presented in Appendix A.3.

Q2: "..different weights for each layer.."

A2: The use of equal weights across the different transformer blocks is motivated by simplicity, aiming to make the design and implementation more straightforward and less complex. This approach also ensures a fair comparison between our proposed attack and other baselines. Optimizing the weights, for instance through a grid search methodology, could enhance the effectiveness of our attack while leaving the baselines un-optimized for this specific task. Nonetheless, using unequal weights would likely improve our attack’s success when aiming for the best performance on its own, rather than in comparison to other approaches. It is also worth noting that the current attack configuration is highly effective, maximizing almost the use of all available tokens.

Q3: "..ensemble strategy performs worse than the single strategy.."

A3: In the ensemble strategy, the adversarial example is trained concurrently across all sparsification techniques, with a different technique randomly selected in each training iteration. In contrast, the single strategy involves training the adversarial example on a single sparsification technique. Intuitively, an adversarial example trained and tested on a single TS technique is likely to achieve the best performance, as it is specifically tailored to that technique. In comparison, the ensemble strategy must adapt the adversarial example to effectively attack multiple techniques simultaneously. Consequently, the ensemble strategy demonstrates a notable ability to affect all sparsification mechanisms to a reasonable extent.

Q4: "..considering the transferability between different ViT architectures.."

A4: Please refer to the general comment.

Q5: "How do the two different components of the custom loss defined in Eq. (3) behave during the training process?"

A5: Both terms decrease as the attack progresses through subsequent iterations, effectively achieving the dual objectives of the attack goal and preserving accuracy.

Q6: ".. random sampling, improve the performance of the proposed ensemble attack.."

A6: Other training strategies could be employed to further improve the performance of the ensemble attack. For example, selecting the worst-performing model at each iteration could enhance the attack's effectiveness on that particular model. However, our goal was to focus on the core concept of the ensemble approach itself, without incorporating additional complexities. We aimed for a straightforward strategy to demonstrate the fundamental effectiveness of the ensemble method.

Q7: "..performance across different ViT architectures.."

A7: In Appendix A.1, we show the performance of our attack on a different size DeiT (tiny) and on the T2T-ViT model. Overall, we observe similar attack performance patterns for the different ViT architectures.

Q8: "..include a brief discussion of the limitations.."

A8: Thank you, we will include a discussion on limitations in the final version.

Thank you for your time, effort and comments.

Q1: "..add some examples and visualizations for the adversarial examples"

A1: visualizations for the adversarial examples, including baselines and our attack variants, can be found in Appendix D. We will also include the perturbations in the final version.

Q2: "..more discussion and derivation on why TS methods are vulnerable to attacks.."

A2: In Section 5.2 "Effect of adversarial perturbations" we discussed several aspects that might explain the success of the attack on each one of the TS mechanisms. Furthermore, in Section 6 "Countermeasures" (lines 390-396) we provide some intuitions and comparison between the different TS mechanisms, highlighting the weak spots of each one. Overall, the main vulnerability that all the TS mechanisms posses is their test-time dynamism and average-case performance assumption, which allows potential attackers to fit in and induce worst-case performance.

Q3: "Will the code and data be publicly accessed?"

A3: Of course, upon acceptance the code and data will be made publicly available.

Thank you for your time, effort and comments.

Q1: "sparsification methods are not robust to adversarial attacks is not surprising"

A1: While it may seem unsurprising now that sparsification methods are not robust to adversarial attacks, this understanding was not trivial prior to our work. Our study provides concrete evidence and detailed analysis, which were previously lacking, thus highlighting the critical vulnerabilities in these methods. By systematically demonstrating these weaknesses, we have paved the way for future research to develop more robust sparsification techniques.

Q2: "The attacks are only partially successful on AdaViT.."

A2: While the success of our attack on AdaViT may seem partial, this is not actually the case. In lines 309-318 in the paper, we discuss this specific phenomena. To summarize, we show that even on a clean image, the distribution of used tokens, layers and blocks across the different transformer blocks are distorted when AdaViT is used. For example, no tokens are used in blocks 4, 10 and 12 regardless of the input sample. In these cases, our attack cannot increase the number of used tokens in these blocks as well. On the remaining blocks, our attack maximizes the number of used tokens, layers and blocks to almost 100%, showcasing that AdaViT is fully vulnerable to such attacks.

Q3: "including more (recent) techniques (e.g. [A]) might further strengthen the paper"

A3: We thank you for mentioning this work. We will include our attack results on this mechanism in the supplementary material final version. For the DeiT-s model, the results for the single-image variant are:

As can be seen from the results, our attack successfully increases the number of GFLOPS from 2.98 (on clean images) to 3.81 (on adversarial images) while maintaining the same level of accuracy.

Q4: "..Are the (class) universal attacks tested on images different..":

A4: The universal attacks tested images are different than those it was trained on, to showcase the perturbation's ability to transfer to unseen images. We will clarify this in the final version.

Thank you for your time, effort and comments.

Q1: "..Compared to methods that do not use TS.."

A1: In theory, the attack's upper bound corresponds to the model's performance when no sparsification is applied, i.e., a "vanilla" model that utilizes all tokens during inference. In practice, in addition to the computational cost of using all tokens, the TS mechanism itself introduces additional overhead, caused by the mechanism's operations, which also influences the performance. While there is an upper bound, our work aims to highlight the potential risk of deploying a transformer model with a token sparsification mechanism. We seek to raise the awareness among users to an attack that can compromise the optimization of the sparsification mechanism.

Q2: "can the three attack methods on ATS, AdaViT and AViT be unified?"

A2: In Section 5 "Transferability and ensemble", we have presented a joint attack in the form of an ensemble, which successfully affected all token sparsification mechanisms simultaneously (see results in Figure 4 penultimate row). In terms of a single loss function that could affect all mechanisms, we could not find any similar characteristics that can be unified and attacked.

Q3: "..report some results on transferability of the same sparsification mechanism on different backbones"

A3: We thank you for your suggestion. This will be clarified in the paper's final version. In addition, we have conducted additional experiments that study the effect of one sparsification mechanism on different backbones. Please refer to the general comment for more details.

Q4: "In Fig.3, what's the density if no TS method is used? is the density 1?"

A4: The results in Figure 3 are evaluated on the DeiT-s model (will be clarified in the figure's caption in the final version). DeiT-s splits the image to $14 \cdot 14 = 196$ patches (tokens), with an additional class token, for a total of 197 tokens. In the case of a vanilla model (no TS method is used), there will just be a vertical line on the 197 value on the x-axis (number of tokens).

Q5: "compared to sponge examples [26].."

A5: In sponge examples, the authors propose attacks for the computer vision and NLP domains. While the results in the NLP domain show excellent performance, this is not the case in the computer vision domain. As reported by the authors in Section 6.2, they were only able to achieve a marginal 1-3% increase in energy consumption. Furthermore, by observing the standard deviation results in Table 3, even this increase in energy consumption does not show a clear trend. Although the paper's authors do not discuss this difference, we hypothesize that it stems from the fact that the vast majority of the computer vision models' activation values are not zero by default, as supported in Phantom Sponges [25].