Fooling the Textual Fooler via Randomizing Latent Representations

Q1: It's really hard to conclude robustness-accuracy tradeoffs from Tables 2 and 3.... Based on the data available, I'd say it's a toss-up whether it's better or worse than AdvFooler.

**A: ** Thank you for the valuable comment. We have revised the performance discussion in Section 4.2, and updated the submission. Indeed, AdvFooler is reliably among the top-3 defenses. However, as mentioned in Section 4.2's discussion, AdvFooler possesses other advantages, such as significantly lower training and inference overhead, being plug-and-play, and requiring no access to the training data.

In our experimments, the calculated the p-values of AdvFooler and the better defenses are below 0.01, indicating that the performance results are statistically significance. We also added this into Supplementary of the revised submission.

We really appreciate the interesting suggestion to "tune SAFER and RunMASK until either the robustness or the accuracy matches AdvFooler". We provide the results when tuning RandMASK to a similar accuracy of AdvFooler. As we can observe, the accuracy under attack of RandMASK decreases tremendously, reducing to half of the original AuA. This shows that, while RanMASK was able to be robust against adversarial attacks, this method also requires larger trade-offs in accuracy compared to other defenses. We added this experiment into Section C.8 (Supplementary), including the experimental setup.

Q2:I think the most compelling case for publication is how lightweight your system is...if you're comparing to needing to tune your noise param for AdvFooler, which while cheap is still some added compute time).

A: In Table 5, Section C.1, we provided the inference time of AdvFooler the AdvFooler and other defenses; AdvFooler only adds a negligible inference overhead to the model, while TMD/RandMASK, SAFER, and ASCC all have non-trivial overhead.

Let N_a be the number of noise scales in AdvFooler. Then, for each noise scale, AdvFooler needs to forward through a small test set. Below, we show the training time for FreeLB and InfoBERT and base-training time + tuning time for AdvFooler (for fair comparison), on AGNEWS. We can observe that, FreeLB and InfoBERT incur significant training overhead, compared to the tuning+base-model training time of AdvFooler.

Q3.1: Generating attacks on the base model in a white box fashion and seeing whether those attacks transfer to AdvFooler after adding noise is still not a white box attack. The attacker needs to be aware and make good use of the specific noise values added (or even the distribution of noises if using some iterative bounding protocol type attack)

Without this added information, the attack you describe is not a white box attack for this setting. It's some sort of attack transferability assessment.

This is again a very useful and insightful suggestion. We updated the paper (Section C.7 in the revised submission) to indicate that these are not truly white-box attacks.

We dubbed this attack "transferred white-box" attacks, due to the fact that the adversary has access to the base model (or even the randomized model), but there still exists a mismatch between the attacker's possessed model and the defender's possessed model due to the added randomization (or the attacker does not make good use of the noise distribution, which is not a designed in the existing white-box approach used for the study), thanks to the reviewer's suggestion. We also believe that developing a true white-box attack that makes good use of the noise values is an interesting and independent extension of our work.

Q6: I very much appreciate your thoroughness and believe there is strong value to this method even if it doesn't beat SOTA. Can you, however, make it even more clear in the contributions that while you achieve close to sota under more constraints, the contribution is not necessarily to achieve SOTA. I think this would also help frame rhe readers perspective early on.

Thank you for the comment. As suggested, we have revised our paper with the following statement in contributions: "The results demonstrate that AdvFooler is a competitive defense, compared to the existing representative textual adversarial defenses, while being under more constraints, including few modeling assumptions, being pluggable, and incurring negligible additional computational overhead."

Q4.1: I'm not sure this experiment is needed for publication, but point taken. The equivalent would be running adp, and then selecting one model within that system at random per query. No further action needed here.

Thank you for the comment.

Q7: I still think the claim of being robust to white box attack is misleading, and this issue is still blocking me from giving a higher rating. I suggest you either frame it as an attack transferability from the base model type of issue instead of mentioning white box attack. Or run a white box attack aware or the noise values added.

We have revised the paper to clarify that this is not truly a white-box attack, in our response to Q6. Again, we appreciate the comment which has helped us improve the clarity of the paper on white-box attacks.

---

Thank you the reviewer for the insightful comments. We hope that your concerns and comments are now addressed. Nevertheless, we're happy to discuss any additional comments you may have!

I still think the claim of being robust to white box attack is misleading, and this issue is still blocking me from giving a higher rating.

I suggest you either frame it as an attack transferability from the base model type of issue instead of mentioning white box attack. Or run a white box attack aware or the noise values added.

Q1: The motivation is not clear. Why does such randomization fool the attackers while not degrading the benign performance?*

A: Thank you for the comment. As explained in Section 3.2 and Theorem 1, the proposed latent feature randomization can fool the query-based attackers since it can mislead the search direction of the adversary's querying process. However, this randomization will affect/degrade the benign performance, a fact we specifically mention in Section 3.3. AdvFooler and other randomization-based defenses, such as RandMask and SAFER, will affect the benign performance. As demonstrated in Figure 4, increasing the noise scale leads to a gradual decrease in benign accuracy; at some point, the large noise scale can even lead to a significant decrease in benign performance. We suggest, in Section 4.3 discussing this figure, that the defender should choose a noise scale for their need (e.g., noise scale corresponds to 1% drop in benign accuracy).

To summarize, we'd like to clarify that AdvFooler's randomization can fool the attackers but will induce a small drop the benign performance. In addition, unlike RandMASK and SAFER, it only has negligible training and inference overhead and can be plugged into any already trained model for better robustness (Table 1).

Q2: Why does AdvFooler can only perplex query-based black-box attacks? It is significant for a defense method to defend against various attacks, such as white-box attacks [1], decision-based attacks [2,3], and so on. It is necessary to validate the effectiveness against these attacks to show the generality of AdvFooler.

A: We'd like to emphasize the practicality of threat model discussed in the paper. For many companies with ML applications, the trained model as well as training data are valuable assets to the companies, thus are secretly guarded. Thus, the attacker rarely has access to the trained model or model’s architecture to perform white-box attacks. Instead, the attacker will rely on querying the models so that they can estimate the optimal perturbation toward adversarial examples, as discussed in Section 1. The same black-box, query-based threat model is frequently examined in several related works [(Nguyen Minh and Luu, 2022), (Zeng et al, 2021), (Dong et al, 2021), (Si et al, 2021), (Li et al, 2021)].

Nevertheless, we also agree that, for broad applicability, it is necessary to validate the effectiveness of AdvFooler against white-box and decision-based attacks, which we already provided in C.7 and C.6 (Supplementary), respectively. These experiments confirm that AdvFooler is still effective against the evaluated white-box and decision-based attacks. As discussed in these experiments, we conjecture that, for white-box attacks, AdvFooler's randomization nullifies the adversarial perturbation and brings it back to the correct side of the decision boundary; since decision-based attacks still involve querying the model, AdvFooler's randomization can return incorrect prediction for some perturbed samples, especially near the decision boundary, and therefore misleads the attacks' optimization process.

Please see our responses below.

Q1: Despite its advantages in terms of simple implementation and minimal computational overhead, AdvFooler's performance falls short of the state-of-the-art. In Table 2, on the AGNEWS dataset, AdvFooler exhibits lower accuracy under attack compared to RanMASK for the BERT-base model, and it also demonstrates lower accuracy under attack than both TMD and RanMASK for the RoBERTa-base model.

We acknowledged, in Section 4.2, that AdvFooler has slightly lower performance than RandMASK and TMD in some experiments.

But, as explained in Section 1, AdvFooler has other important advantages compared to RandMASK and TMD: (1) having negligible computation overhead while (2) simultaneously achieving comparable robustness to RandMASK and TMD, and (3) being plug-and-play, which allows it to be used without modifying existing parameters of the model. Simultenously achieving (1) and (2) is an extremely challenging task; for example, TMD and RandMask require significant additional training and inference overhead for similar robustness to AdvFooler. Also for, for RandMASK, its clean accuracy drops significantly (>4% for RandMASK vs. < 1% for AdvFooler). For these reasons, AdvFooler is significantly more practical for usage in real-world applications, a significant contribution to the adversarial defense domain.

Q2: The selection of the hyper-parameter for noise scale in AdvFooler is not entirely clear. The authors claim that they choose the $\nu$ value based on the criterion that the clean accuracy drops by at most 1% using the test set. However, it seems not the case in Table 2 and 3. In Figure 4, the curves of AuA are not monotone. Do authors also consider AuA values when choosing noise scale?

A: We'd like to, first, re-iterate the discussion in Section 3.3: "the defender can select the noise scale $\nu$ at which the variance causes the clean accuracy drops by a chosen percentage (e.g., in our experiments, it is 1%)". In practice (as discussed in threat model), the defender only has the trained model and a small clean test set and makes no assumption about the attacks; using the clean test set, defender finds the largest noise scale corresponding to a certain, tolerable drop in clean accuracy. For example, given the values we used to plot Figure 4 as in the table below, starting with the original clean accuracy of 95.14%, the defender would choose $\nu=0.6$, which corresponds to 94.57% clean accuracy, assuming a selection of 1% drop in performance. For 2% drop (or 94.57% clean accuracy), $\nu$ is set to 0.80. As we can see, the larger $\nu$ generally increases the robustness (43.90% and 48.80% AuA for $\nu=0.6$ and $\nu=0.8$, respectively, on TextFooler); however, when $\nu$ is too large, as we can observe in this figure, it will also interfere clean accuracy (a sudden, significant drop in clean accuracy) and correspondingly decrease the robustness of the model as well.

Q3: Instead of adding noises to all layers by the same noise scale, what would the results be if adding different noise scales to different layers?

A: Thank you for the interesting question. We provide the experiments (AGNEWs) for AdvFooler with different noise scales added to different layers (we multiply the original noise scale by the standard deviation of that layer’s output)

The results show that this variation has good performance compared to our current AdvFooler’s version.

Please see our responses to your comments below:

Q1: Could the authors compare and contrast their approach to applying DP-style noise to the embeddings generated by the hidden layer?

A: Thank you for the comment; however, we’re not sure what DP stands for and will assume DP refers to Differential Privacy, a privacy protection framework that introduces a small change in the data for a guaranteed, bounded change in the distribution of the algorithm’s outputs. DP-style defenses such as (Lécuyer et al 2019) certify the defense under the DP framework; it is important to note that SAFER and RandMASK, evaluated in the paper, are also certified defenses and share a similar approach to DP-style defenses. DP-defense and RandMASK still require training overhead to tune the added noise so that it does not interfere with the model's prediction. On the other hand, AdvFooler adds noise to the latent features at inference time to misdirect the attacker's querying process.

We have added this discussion to our revised submission.

Lécuyer, M., Atlidakis, V., Geambasu, R., Hsu, D.J., & Jana, S.S. (2018). Certified Robustness to Adversarial Examples with Differential Privacy. 2019 IEEE Symposium on Security and Privacy (SP), 656-672.

Q2: Results from Table 2 suggest that their approach is not the very best compared to other approaches. What are the merits of the proposal then? Apart from reduced computational overheads?

A: As explained in Section 1 (especially Table 1), AdvFooler (1) reduces significant computation overhead while (2) simultaneously achieving comparable robustness to other defenses. This is an extremely challenging task; for example, TMD/RandMask/SAFER or adversarial training approaches require significant additional training overhead for similar robustness to AdvFooler; TMD/RandMask/SAFER also incurs non-trivial inference overhead, making them less practical in real-world application. Another advantage of AdvFooler is (3) plug-and-play, as also indicated in Table 1, which allows it to be used without modifying existing parameters of the model. Finally, as indicated in Section 3.3 and 4, AdvFooler allows the user to (4) control how much accuracy they want to trade-off for robustness using a small test set. (1), (2), (3) and (4) make AdvFooler significantly more practical to employ in real-world applications, a significant contribution to the adversarial defense domain.

Q3: The authors empirically note that adding small amounts of noise to the logins does not change the prediction. But this is not fundamental nor clear why this is the case. Could the authors elaborate?

A: Defenses via adding noise (including AdvFooler and RandMASK/SAFER) will inevitably affect the prediction or clean accuracy. While RankMASK/SAFER perturbs the input and require additional training and/or inference overhead to tune the added noise for a small accuracy drop, AdvFooler adds noise to the hidden features only at inference time, an approach which is theoretically shown to be able to fool the attacker (discussed in Section 3.2). To choose this noise, however, we suggest using a small test set and finding a large-enough noise for a specific tolerance of accuracy drop (discussed in Section 3.3 and Figure 4). This allows the defender to control how much accuracy they want to trade off and experiments show that our approach can achieve comparable robustness with other defenses while having only <1% accuracy drop.