TROJFSL: TROJAN INSERTION IN FEW SHOT PROMPT LEARNING

We thank reviewer P9fg for their careful and valuable review of our manuscript and for providing constructive feedback.

Q1: This paper only considers syntactic triggers. However, the generality of the proposed method with respect to different trigger types remains underexplored. If the method is indeed trigger-agnostic, it is imperative that an evaluation is conducted to demonstrate its effectiveness across a broader spectrum of triggers.

In Table E, we investigate various triggers, including "cf," "bb," and "oq." It is evident that our TrojFSL consistently exhibits high CDA and ASR across different triggers.

Table E: Study of different triggers.

Q2: The authors claim that the target class is susceptible to receiving a larger number of input samples compared to other non-target classes, subsequently leading to a low CDA. Intuitively, this can be balanced by setting the poisoning ratio. Setting an appropriate poisoning ratio can achieve a good CDA and ASR.

The setting is within few-shot scenarios, specifically utilizing 16-shot samples in our experimental setup. As depicted in Figure 2(a), with the increase in the number of poisoned samples (same as to increase the poisoning ratio), the CDA decreases while the ASR increases. The reason for this is that before an attack, the target class and a non-target class possess an equal number of samples. However, once poisoned samples are added to the target class, it results in a larger sample count for the target class. This imbalance is particularly pronounced in the case of few-shot samples. Furthermore, there isn't an ideal poisoning ratio that can effectively maintain a high CDA (Clean Data Accuracy) while keeping the ASR (Attack Success Rate) low. A large poisoning ratio will cause an accuracy decrease; A small ratio will not have a high ASR.

Q3: The paper falls short in elucidating some of the experimental settings, particularly when benchmarking TrojFSL against previous works. The absence of a detailed experimental setup undermines the reproducibility and the clarity of comparative analysis. Additionally, there is a noted inconsistency with the results of the referenced paper BadPrompt [1], where BadPrompt reportedly attains a 100% ASR with merely two poisoning examples on SST-2. An explanation of this discrepancy, along with a thorough delineation of the experimental setup, would bolster the comparative narrative.

In the BadPrompt paper, the assumption is that the attacker can tune the pre-trained models (PLMs). In contrast, our setting assumes that the PLMs are frozen, making it more practical. Consequently, BadPrompt performed in the frozen PLMs exhibits inferior performance. We conducted this Frozon-PLM Badprompt to have a fair comparison with our work. We will make our codes available for reproduction.

[1] Cai, Xiangrui, et al. "Badprompt: Backdoor attacks on continuous prompts." Advances in Neural Information Processing Systems 35 (2022): 37068-37080

Q4: The discussion on defense strategies is somewhat not enough. While the authors posit that RAP and ONION are ineffectual against TrojFSL when utilizing invisible syntactic triggers, the efficacy of these defenses under alternative trigger patterns employed by TrojFSL remains unexplored. Moreover, the consideration of elementary adaptive defenses, such as rephrasing the input, could offer a more comprehensive insight into the defense landscape against the proposed attack.

We concur with your viewpoint that RAP and ONION prove ineffective against TrojFSL when employing invisible syntactic triggers. This enables us to circumvent the detection mechanisms of RAP and ONION, facilitating the implementation of backdoor attacks. We appreciate the reviewer's suggestions regarding rephrasing the input. It's noteworthy that the effectiveness of defense strategies relies on defenders having knowledge of specific syntactic structures to counter the attacks; otherwise, the backdoor attacks persist.

Q5: Is the attack trigger-agnostic?

Yes, the attack is trigger-agnostic. Same as Q1.

We thank reviewer x2RS for their careful and valuable review of our manuscript and for providing constructive feedback.

Q1: Lack of comparison with baselines. The proposed method is only compared with the baselines on one dataset for one model. The performance of the baseline is clearly worse than the results reported in the original paper (e.g. for PPT).

In Table C, we compare our TrojFSL against prior backdoor attacks on SST-5 and MR datasets. We can see our TrojFSL has the optimal CDA and ASR. We have incorporated the results into Table 3 in the updated version of the paper.

Table C: The comparison under the setting of frozen PLM and 16-shot learning.

The results in the PPT paper are derived from analyses of comprehensive training downstream datasets, encompassing tens of thousands of samples. However, for an equitable comparison with our work, we revisited these studies with a focus on few-shot (e.g., 16-shot) learning contexts. Specifically, the PPT paper operates under the assumption that an attacker has access to a user's entire downstream task dataset, which facilitates optimal performance on the original task. Take the SST-2 dataset as an example, where it has 60k samples and a poison ratio of 0.1; the PPT assumes complete dataset access for the attacker. In contrast, Table 3 is based on the premise that the attacker only has access to 16-shot samples, a significantly smaller subset of the full dataset. This leads to less impressive results for PPT in such a constrained scenario. Moreover, when we expand the number of few-shot samples in Table D, there is a notable enhancement in both CDA and ASR, underscoring the influence of an increased sample count.

Table D: Study of PPT's few-shot number

Q2: Omission of existing work, DecodingTrust. Incorrect statement: "No prior prompt-based backdoor can be implemented via few-shot prompt-tuning with frozen PLMs".

First, thanks for pointing out this concurrent paper. According to ICLR reviewer guidance: "if a paper has been published on or after May 28, 2023, there is no obligation for authors to compare". The DecodingTrust [1] paper was made available online on 2023-06-20.

We cited this DecodingTrust paper in Introduction section and clarify it is not for prompt-tuning scenarios, instead, it tests the attacks of hand-crafted engineered prompts on GPTs. Notably, we introduced the importance of prompt tuning attacks in the paper.

Our assertion that "no prior prompt-based...via few-shot prompt tuning with frozen PLMs" is specific to few-shot prompt tuning attacks. For clarity, we've modified this to "no backdoors on prompt-tuning with frozen PLMs on few-shot downstream samples." This revision emphasizes prompt tuning attacks, distinguishing them from other types of prompt injection or prompt engineering attacks.

Q3: The experiments regarding weight balancing on SST-5 with more classes

Table B shows that ( \beta = 0.5 ) and ( \alpha = 1 ) work well on the SST-5 dataset.

Table B

Q4: Other Incorrect statements." the adversary.... change their labels to the target class" given there is a clean-label backdoor attack and a handcrafted backdoor.

Our statements pertain to the context of few-shot prompt tuning, where the attacker is limited to accessing only a few-shot sample size, and the PLMs remain fixed. In such a scenario, executing clean-label attacks and creating effective handcrafted backdoors remain formidable challenges due to the constraints of limited data samples and the immutability of the PLMs.

In the updated paper, we revised the statement to “In the context of few-shot prompt-tuning, the adversary's popular strategy involves collecting input samples from non-target classes and relabeling them as the target class."

Q5: Typos

Fixed.

We thank reviewer oqR6 for their careful and valuable review of our manuscript and for providing constructive feedback.

Q1: Technical innovation.

As Table 1 shows, directly applying prior work to a few-shot prompt attack suffers from either low ASR or low accuracy. Identifying new techniques that can simultaneously achieve high ASR and maintain high accuracy presents a significant challenge. To design techniques, it is very important to understand the reasons why a low ASR and low accuracy happen. Thus, our technique novelties not only include the formulation of identified issues: 1. imbalanced poisoned dataset, 2. overfitting, and 3. lack of attention awareness, but also include their corresponding solutions: 1. Balancing the Poisoned Dataset, 2. Selective Token Poisoning, and 3. Trojan-Trigger Attention. These three technique solutions have been developed based on our insights into the inherent characteristics of the issues they address. For example, before an attack, the target class and a non-target class possess an equal number of samples. However, once poisoned samples are added to the target class, it results in a larger sample count for the target class. This imbalance is particularly pronounced in the case of few-shot samples. And it is not straightforward to balance the poisoned samples for few-shot prompt attacks. Tuning the poisoning ratio cannot solve the problem. We formulate the method into a simple equation 2. We convert a complex problem into a simple format. Similarly, our selective token poisoning is designed to only insert Trojans into the tokens in the soft prompt with the lowest importance score, while the other tokens in the prompt remain untainted. This is a backdoor-prompt co-design to solve the attack overfitting issue. The trojan-trigger attention is designed to increase the attention on triggers instead of clean tokens and prompts, which is novel and significant for enhancing clean accuracy and ASR.

Q2: Excessive hyperparameters requiring adjustment. For instance, adjusting beta and alpha in the balanced dataset.

Most parameters already exist in prior backdoor works such as the poison ratio $\alpha$ and the coefficient $\lambda$. The parameters we introduced in the paper are dataset balance coefficient $\beta$, pruned token number $\gamma$, and attention-loss coefficient $\lambda_1$. Notice that $\beta$ is fixed once poison ratio $\alpha$ and the weight coefficient $\lambda$ are known since our balanced dataset in Equation 2 requires $\beta +\lambda \alpha(n-1)=1$, where $n$ is the class number. In our experiments of Table 2,3,6,7, $\beta=0.5$, $\gamma=1$ and $\lambda_1=1$ (We have added it in section 4). One can readily apply this set of parameters in experimental setups. It is important to note that we have conducted detailed studies on these three parameters: Tables 4 and 5 analyze the parameter $\beta$, Table 8 is dedicated to the pruned token number $\gamma$, and Table 9 explores the attention-loss coefficient $\lambda_1$.

Q3: Black-box setting for APIs? If not, please discuss.

White-box model attacks are still important. Numerous model-sharing platforms, such as Hugging Face and Model Zoo, facilitate model submissions and downloads for users. Consequently, the exploration of white-box attacks remains crucial. Notably, our work on few-shot prompt attacks addresses a previously unexplored area and white-box few-shot prompting attacks still suffer from several challenges. We identified several unveiling challenges and proposed solutions to solve them, which marked a significant stride in the direction of few-shot prompt attacks.

Q4: More diverse combinations of techniques and Hyperparameters on more representative datasets.

We examined various combinations of the proposed techniques in Table A to further show the individual impact of each technique. It is evident that with the inclusion of any proposed technique, both CDA and ASR show an increase.

Table A: Various Combinations of TrojFSL Techniques.

Table B shows that $\beta = 0.5$ and $\alpha = 1$ works well on the SST-5 dataset too.

Table B