Backdoor Attacks in Token Selection of Attention Mechanism

We appreciate the reviewer's interest and recognition of our contributions and the novelty of our work. We will correct the typo in the final version.

Regarding the connection with practical settings, we have some conjectures about possible defense mechanisms. Suppose that the learner has knowledge of the relevant tokens. In that case, a simple sanity check could be performed: after training the transformer by optimizing only the tunable token $\mathrm{p}$, one can examine whether $\mathrm{p}$ exhibits a strong correlation with signals that are not relevant tokens. If such a correlation exists, it may indicate that the transformer has been compromised by a backdoor attack.

Since optimizing $\mathrm{p}$ is equivalent to optimizing $\mathrm{W}$, our results suggest that backdoor triggers are injected into the key and query matrices. Therefore, another potential defense strategy in practice would be to apply dropout layers for the attention model. By randomly masking out poisoned neurons in the key or query matrices, dropout could introduce inconsistencies in the model’s output if the model has been poisoned. One can also adopt the idea from [1] to detect poisoned data within the training sample by identifying cases where a small proportion of the extracted features differ significantly from the rest. These are immature ideas and require thorough investigation, particularly in the context of practical transformer architectures, which is beyond the scope of this paper.

[1] Tran, Brandon, Jerry Li, and Aleksander Madry. "Spectral signatures in backdoor attacks." NeurIPS 2018.

We appreciate the reviewer's interest and recognition of our contributions and the novelty of our work.

W1: The lower bound on the number of iterations $\tau_0$ depends on the proportion of relevant tokens $\zeta_R$, the proportion of irrelevant tokens $\zeta_P$, the number of tokens $T$, and the strength of poisoned signal $\alpha$. $\tau\geq \tau_0$ is required to guarantee that, for any give $\epsilon>0$, the softmax probability of the relevant token is at least $1-\epsilon$ for the standard training sample, and the softmax probability of the poisoned token is at least $\frac{1}{|\mathcal{P}|}-\epsilon$ for the poisoned training sample. Such condition can be met due to Lemma 5.1. The proof of generalization guarantee requires $\epsilon\lesssim \min(\frac{T}{(\frac{1}{\zeta_R}-1)^4}, \frac{1}{(\frac{1}{\zeta_P}-1)^{\frac{4}{\alpha}-1}})$. Smaller $\epsilon$ leads to larger $\tau_0$.

W3: We assume that the union of $\mathcal{P}$ and $\mathcal{R}$ is an empty set to guarantee that adding poisoned tokens does not alter the semantic meaning of the original input. Intuitively, if a poison pattern modifies an image in a way that changes the original object, or if a modified word changes the semantic meaning of the sentence, the classifier should not be expected to predict the original label. We will include a remark on this in the final version.

We will unify the use of punctuation of formulas as suggested by the reviewer.

We appreciate the reviewer's interest and recognition of our contributions and the novelty of our work.

Q1: Regarding the orthogonality assumption, such assumption can be relaxed to the setting where the relevant signals $\mu_{\pm 1}$ and the poisoned signals $\tilde\mu_{\pm 1}$ are correlated, and our proof still holds with minor modifications. We discuss this on page 6, left column, lines 295–303. We can add more clarification in our final version.

Q2: We conduct several experiments using the same synthetic dataset as described in the paper. We choose $|\mathcal{R}|=|\mathcal{P}|=1$ and vary $\beta$ across $0.1,0.2,0.3,0.4$ and $\alpha$ across $1,2,3,4$. We compare the poison accuracy between jointly optimizing $\nu$ and $\mathrm{p}$ versus optimizing only $\mathrm{p}$ under the same $\alpha$ and $\beta$. Our results show that while the final poison accuracy is similar in both cases after sufficient training iterations, joint optimization leads to a faster convergence rate. We hypothesize that jointly optimizing $\nu$ and $\mathrm{p}$ may strengthen the backdoor attack in more practical scenarios, such as when training on a more complex dataset or using a more sophisticated attention architecture. Understanding the effects of joint optimization remains an interesting research direction, which we highlight as a future avenue in our paper (page 6, right column, lines 322–324).

Q3: We didn't run experiments on transformers pre-trained on large corpora.

Q4: This is an excellent question. We have some conjectures about possible defense mechanisms. Suppose that the learner has knowledge of the relevant tokens. In that case, a simple sanity check could be performed: after training the transformer by optimizing only the tunable token $\mathrm{p}$, one can examine whether $\mathrm{p}$ exhibits a strong correlation with signals that are not relevant tokens. If such a correlation exists, it may indicate that the transformer has been compromised by a backdoor attack.

Since optimizing $\mathrm{p}$ is equivalent to optimizing $\mathrm{W}$, our results suggest that backdoor triggers are injected into the key and query matrices. Therefore, another potential defense strategy in practice would be to apply dropout layers for the attention model. By randomly masking out poisoned neurons in the key or query matrices, dropout could introduce inconsistencies in the model’s output if the model has been poisoned. One can also adopt the idea from [1] to detect poisoned data within the training sample by identifying cases where a small proportion of the extracted features differ significantly from the rest. These are immature ideas and require thorough investigation, particularly in the context of practical transformer architectures, which is beyond the scope of this paper.

[1] Tran, Brandon, Jerry Li, and Aleksander Madry. "Spectral signatures in backdoor attacks." NeurIPS 2018.

Regarding missing references: we have discussed some of the theoretical work on multi-head attention in Section 2, page 2, right column, lines 72-78. We will include more references in our final version.

We appreciate the reviewer's interest and recognition of our contributions and the novelty of our work. We acknowledge that our current results rely on restrictive assumptions and that our experiments serve primarily as a proof-of-concept for our theoretical findings. We have discussed these limitations in the paper. As this is the first work to address how backdoor triggers influence the optimization of attention-based models, our goal is to provide valuable insights into this problem. We agree that refining and relaxing these assumptions is an important direction for future research.