Understanding the Robustness of Randomized Feature Defense Against Query-Based Adversarial Attacks

Please see our responses to your comments below:

Q1: Assumption 1, Figure 1 and 2 are not referred to in the main text. It is confusing on the purpose of presenting the assumption 1 and Figure 1 and 2.

A: Thank you for the observation. We have made incorrect references for Figures 1 and 2, when including the Supplementary File within the Main paper. Please see our clarifications below:

• ‘Figure 3’ in Section 3.3 should be ‘Figure 1’,
• ‘Figure 5’ in Section 3.5 should be ‘Figure 2’,
• The discussion for Assumption 1 is immediately above its declaration. It allows us to ensure that small noise diffuses but does not shift the prediction, or adversarial samples of the original model are adversarial samples of the randomized model. We have updated this discussion in the revised submission.

Q2: The assumption and the theorem are incorrect. Even when the noise is small, the expectation of the randomized model is not necessarily consistent with the original model on the same data point, one simple counter-example is that when the input x is at the decision boundary, a small perturbation can change the prediction, so small noise may change the prediction.

A: As discussed in Section 3.2, given Assumption 1, adversarial samples of the original model are adversarial samples of the randomized model, on expectation. In other words, at the decision boundary, while a specific sample of the small Gaussian noise may change the prediction, but expectedly, it should be the same prediction as that of the original model (across several samples of the small Gaussian noise). This assumption is typically made in other relevant randomization works, including the evaluated RND defense.

Q3: Theorem 1 is based on incorrect derivation, Eq. (23) and (24) may be incorrect as the gradient $\nabla_{h(x)}(L \cdot g)$ is anisotropic so the multiplication with Gaussian noise should not be an i.i.d. Gaussian noise.

A: Thank you for the comment. Please see our response below, confirming that it is still indeed an i.i.d. Gaussian noise.

For a random vector $x$ following $\mathcal{N}(\mu, \Sigma)$, we have $Mx\sim\mathcal{N}(M\mu, M\Sigma M^T)$ which is still Gaussian noise. In Equation 23, since $\delta_1 - \delta_2\sim\mathcal{N}(0, 2\nu I)$ and $\nabla_{h(x)}(L \cdot g)2\nu I \nabla_{h(x)}(L \cdot g)^T=2\nu ||\nabla_{h(x)}(L \cdot g)||^2_2$, we have $\nabla_{h(x)}(L \cdot g)(\delta_1 - \delta_2)\sim\mathcal{N}(0, 2\nu ||\nabla_{h(x)}(L \cdot g)||^2_2)$. Since the noise added by the attack and the defense at each application are independent, $\nabla_{h(x)}(L \cdot g)(\delta_1 - \delta_2)$ follows i.i.d Gaussian noise.

A similar derivation is applied to Equation 24.

Q4: In addition, the assumption of the proof is the value of \nu and \mu are small, so the approximation holds, but in the experiments, the value of \nu is not present, and the value of \mu is as large as 0.3, which is not negligible.

A: In our experiments, as indicated (e.g., Tables 1 and 2), the selected $\nu$ corresponds to a small drop in accuracy (1-2%). These $\nu$ values are still very small, with magnitude less than 0.016. Our experiments show that the user can control the degree of clean accuracy loss for their application when selecting $\nu$, and typically, the user will not use a large $\nu$ value that will cause significant degradation in clean accuracy.

On the other hand, $\mu$ is the step size that the attacker uses to search for the correct perturbed direction. For Square attack, the maximum square size is 0.3, however, the added noise is still bounded by the $\ell_{\infty}$ ball of radius 0.05. For NES $\mu$ is the step size of finite difference method, which should be small so the attacker can obtain a good approximation for the gradient; when $\mu$ is too large, this approximation becomes unreliable. In the experiments, we run the attacks with small exploration step size, in particular for Square attack 1 and 0.005 for $\ell_{2}$ and $\ell_{\infty}$ NES attacks, respectively. Note that, these values are still small, to ensure gradient approximation is reliable. These exact values have also been used in previous works, including RND and SND.

Please see our responses to your comments below:

Q1: randomized output is stochastic and unpredictable... a randomized model entails all predictions have some unaccounted chance of being wrong. average the outputs across multiple runs, but doing so would reduce the robustness and just converge back to the deterministic case (and with increased costs).

A: Randomized models have been extensively studied in the literature as an effective way to defend against adversarial attacks (Liu et al. 2017; He et al. 2019; Cohen et al. 2019; Salman et al. 2019; Byun et al. 2021; Qin et al. 2021). Indeed, since randomization may cause unpredictable predictions, existing works require training and model ensembles, both of which are expensive, to ensure minimal impact of the noise (i.e., cause a decrease in clean accuracy), while achieving high robustness. In contrast, our defense, and also RND, allow the user to control how much random noise to add to ensure a minimal drop in clean accuracy (using a small test set). For example, our defense achieves high robustness in Tables 1 and 2 against several attacks while only causing 1-2% accuracy drops in performance. Please note that defense approaches such as adversarial training cause a significant drop in clean accuracy (e.g., 10% or more in ViT (Mo, Yichuan, et al)) to achieve good robustness.

Mo, Yichuan, et al. "When adversarial training meets vision transformers: Recipes from training to architecture." NeurIPS 2022.

Q2: Definition of a successful attack. I argue that in security-sensitive applications (e.g., authentication, malware detection, etc.), it is enough for the model to be fooled once

A: Our empirical experiments focus on evaluating whether the attack can truly find adversarial examples. For non-randomized models, when an attack arrives at the decision that $x$ is an adversarial example, $x$ unquestionably is on the other side of the decision boundary. However, for a randomized model, $x$ could be still on the correct side of the decision boundary, but the added randomization shifts it to the other side of the decision boundary; thus, in principle, $x$ is still not an adversarial example, and if we use 1 single application for evaluation, an attack could be lucky or a randomized defense could be unlucky. Consequently, for a fair evaluation of the effectiveness of a defense, we forward $x$ multiple times and decide that it is an adversarial example if the majority of the results say so, as seen in our paper.

Nevertheless, we understand that in the case where we ignore fair evaluation, the attack is allowed to be lucky and just has to fool the defense once, as given in the example in the comment. Therefore, we also provide the experiments when forwarding $x$ only once, as below. As we can observe, our defense is still effective against the attacks.

Robustness with a single trial, 10k queries, 1%-accuracy noise scale, ImageNet

Q3: All the analysis in the paper uses the first-order Taylor approximation. An empirical evaluation of the approximation error would make all the analyses more convincing.

A: Thank you for your suggestion. We include the histogram of the approximation error of ViT on ImageNet in Section F.1 (Supplementary). As we can observe, the error is relatively small.

Q4: First, I wonder which intermediate layer is picked for all the results in Section 4. empirically or according to some metric?

We pick the penultimate layers of the base models for randomization, in all experiments. We have shown with theoretical analysis in Figure 1 (main) and empirical analysis in Table 12 (supplement), that the randomizing at a deeper layer consistently achieves the best robustness; for example, randomizing the last layer in VGG or ViT achieves 8-10% improvement over the previous layer. We have already revised the paper (Section 4.1) to clarify this discussion accordingly.

Q5: The authors have taken some steps to adapt the attacks for the randomized defense, mimicking obvious modifications that the real adversary might do (EoT and Appendix D.4). I really like these initiatives and also wonder if there are other obvious alternatives. One that comes to mind is to use attacks that maximize loss given a fixed $\eps$ budget. These attacks should not have to find a precise location near the decision boundary which should, in turn, make it less susceptible to randomness.

This actually does NOT mean that the randomness is not beneficial. Suppose that the loss-maximizing attack operates by estimating gradients (via finite difference) and just doing a projected gradient descent. One way to conceptualize the effect of the added noise is a noisy gradient, i.e., turning gradient descent into stochastic gradient descent (SGD). SGD convergence rate is slowed down with larger noise variance so the adversary will have to either use more iterations or more queries per step to reduce the variance. Either way, the attack becomes more costly. I suggest this as an alternative because it directly tests the benefits of the added noise without exploiting the fact that the distance-minimizing attacks assume deterministic target models.

A: Thank you for your appreciation of our analysis and indeed a great observation regarding the fixed budget. In fact, NES, Square attack, and SignHunt, all of which are evaluated in our paper, work in the principle of maximizing loss given a budget. Specifically, NES approximates the gradient directly and applies projected gradient descent, while Square attack and SignHunt search for the adversarial sample at the boundary of the $\ell_p$ ball.

Q6: additional figures: Scatter plot of the robustness vs the ratio in Theorem 1. Scatter plot of the clean accuracy vs the ratio in Eq. (14). Robustness-accuracy trade-off plot

Thank you for your suggestions. In fact, the robustness averaged over the test samples at each layer, corresponding to Figures 1 and 2, are provided in Table 12 (Supplementary). We can observe an increase in robustness as the ratio becomes higher at deeper layers. We have updated these Figures, in the revised paper, with the performance results created in Table 12, for a better explanation.

As discussed in Section 4, our defense allows the user to choose $\nu$ that results in a “controlled” decrease in clean accuracy, using a small test set. For example, we provided the results of varying $\nu$ in Tables 1 and 2 that correspond to 1%, 2%, 3% and 4%. drops in clean accuracy. In general, increasing $\nu$, or more randomness, will lead to a decrease in clean accuracy but also an increase in robustness, as observed in these tables. In addition, we provide the robustness (Accuracy under Attack) when varying $\nu$ (corresponding varying drops in clean accuracy) on CIFAR10/VGG19/Square Attack setting: AuA = [56.2, 58.4, 62.3, 62.8, 63.5] Acc = [95.33, 94.94, 94.55, 93.58, 92.09]

Q7: The most important contribution is the analysis on the gradient norm (i.e., sensitivity of the model) of benign and perturbed samples. The proposal to add noise to the intermediate layer instead of the input in itself is relatively incremental. However, the theoretical analysis does seem particularly strong to me, even though it does build up a nice intuition of the scheme. This is a minor weakness to me personally, and I would like to see more empirical results, as suggested earlier

A: Thank you for appreciating our work. Here, we summarize our responses to the earlier questions:

• Evaluation with a single application on the successful query (answered in Q2, added to the revised paper)
• Error analysis of first-order approximation (answered in Q3, added to the revised paper)
• Performance at different layers with the same drop in accuracy (answered in Q6, provided in Table 12 of original submission)
• Robustness-Accuracy tradeoff (answered in Q6 with additional experiments) Different from existing randomized-feature defenses, which additionally require adversarial expensive training and ensemble, our work opens the possibility of randomizing the latent layers at inference time without these hassles (being lightweight and plug-and-play) and with a strong theoretical foundation. In addition, unlike randomizing the input, such as RND, our defense gives the user protection against non-continuous input, such as graph data; however, this is beyond the scope of our paper and deserves another independent study.

Please see our responses to your comments below:

Q1: The motivation to inject feature noise is not clear compared to injecting input noise. "Unlike previous randomized defense approaches that solely rely on empirical evaluations to showcase effectiveness" is not correct, since RND also provides lots of theoretical analysis as the authors acknowledged in Sec. 2.3. The results are not significantly better than RND, but injecting feature noise requires a careful choice of the layer.

A: Thank you for the suggestion. In the above sentence, we referred to the lack of rigorous theoretical analysis of the robustness against black-box, query-based attacks when randomizing the internal features of the model. Similar to our work on the feature space, RND indeed provides a theoretical analysis on the input space. We have already revised the paper (Section 1) to clarify this claim accordingly.

As we can observe in Tables 1 and 2, our randomized feature defense consistently outperforms RND (in several cases, the improved robustness of our defense is almost 10-20% more than that of RND) for all evaluated attacks, except NES, where RND occasionally performs better than our method (in these cases, RND’s slightly better robustness of ~2-3%).

Finally, regarding the comment that “our method requires careful choice of the layer”:

• We have shown with theoretical analysis in Figure 1 (main) and empirical analysis in Table 12 (supplement), that the randomizing at a deeper layer consistently achieves the best robustness; for example, randomizing the last layer in VGG or ViT achieves 8-10% improvement over the previous layer.
• In fact, all our experiments randomize only the penultimate layers of the models. Consequently, the complexity of using our method is equivalent to that of RND. We already provided clarification on this claim (Section 4.1) in the revised submission.
• Please note that an advantage of our defense over RND is that randomizing an internal layer makes the method suitable for discrete data, such as graphs. However, this is beyond the scope of our paper and deserves an independent study.

Q2: 1. The idea of injecting noise into hidden features is not novel, seeing Parametric Noise Injection: Trainable Randomness to Improve Deep Neural Network Robustness against Adversarial Attack, CVPR 2019. Although this is for defending against white-box attacks, adopting it for black-box attacks does not seem a significant contribution.

A: Although injecting noise to the features is previously proposed (such as in the mentioned paper), these methods require additional training to tune the noise, which is computationally expensive, especially when dealing with large and complex datasets. In contrast, our defense, and also RND, are lightweight, plug-and-play and can be applied to any pretrained model without additional model training. Our defense is effective against black-box attacks because it is designed to fool the querying process of these attacks, as seen in our theoretical and empirical analysis.

Q3: Does the proposed method have an advantage against AAA in defending score-based attacks? AAA is not designed for decision-based attacks, where the authors use AAA for comparison.

A: We already provided the experiments, comparing AAA to our defenses on score-based attacks in Section E.3 (Supplementary). The experiments show that that our defense improves the robustness and has comparable results to AAA on score-based attacks. However, for decision-based attacks, as provided in Section 4.3, our defense has superior performance. From a practical perspective, a defense should work well against many types of query-based black-box attacks, and these experiments show that our defense has more practical utility than AAA.

Please see our responses to your comments below:

Q1: I understand that the paper focuses on black-box attacks, but in the experiment section, the authors may try evaluating models with white-box attacks as well.

A: Our work focuses on studying the effect of randomized models on black-box attacks, which are more practical in MLaaS system. In these systems, the adversary rarely has access to the trained model or model’s architecture to perform white-box attacks. Although our analysis does not cover white-box attacks, in Section E.4 we still conduct experiments for white-box attacks on randomized feature defense. The experimental results on C&W and PGD show that our defense can also boost the model’s robustness against white-box attacks, which demonstrates the broad utility of our method beyond black-box attacks.

Q2: Please check the parentheses in equation (7).

A: Thank you for the comment. We have fixed the typo in the revised submission.

Q3: In page 5, can you please give a reason for this sentence "We can observe that these ratios become higher when the data are perturbed toward the adversarial samples. In other words, the randomized model is more robust during the attack."?

A: In our theoretical analysis, we show that the robustness to black-box attacks is controlled by three terms: the scale of noise vector added by the attack $\mu$ and the defense $\nu$, and the ratio of the norm of the gradient with respect to the feature where noise is added and the input $\frac{||\nabla_h(L\circ g)||}{||\nabla_h(L\circ g)||}$. While the scales of the noise vectors are fixed during an attack, the last term is dependent on the input and where in the model the defense is performed. During the attack, the input is perturbed and thus changes the above ratio if noise is injected into hidden layers of the model. Figure 1 illustrates that the ratio increases, especially for deeper randomized layers, when the attack happens, implying that randomized feature defense has a higher chance than input defense to fool black-box attacks.