DiffHammer: Rethinking the Robustness of Diffusion-Based Adversarial Purification

Thank you for the valuable feedback, here's our response to the concerned problems.

Advantages and costs of N-evaluation

Summary: We were the first to propose using $N$-evaluation to enhance evaluation accuracy and improve attack effectiveness and efficiency. In terms of evaluation, we demonstrate that 1-evaluation underestimates resubmission risk. An appropriate $N$ (e.g., 10) is acceptable to attackers, making the risk realistically alarming. We'll include a discussion of this section in the paper.

In terms of attack, we leverage the byproduct of $N$-evaluation, which incurs minimal additional overhead compared to the $N$-EOT attack. Our algorithm only require a small $N$ to overcome the gradient dilemma for sufficient attacks.

Current robustness evaluations typically rely on 1-evaluation. However, in diffusion-based purification, the attack success rate for about 50% of samples ranges between 0 and 1, suggesting that attackers can undermine defenses by resubmitting. Evaluating resubmit attack risk depends on estimating attack probability, which 1-evaluation's non-zero-or-one result fails to provide. Even for samples with a 10-20% success rate (around 20% in our experiments), attackers can expect more than one successful attack out of 10 resubmissions with a feasible cost. Our analysis shows that 1-evaluation underestimate this risk statistically, as discussed in Section 3.1. Therefore, our proposed $N$-evaluation is practical amid the rise of diffusion-based defenses.

Another contribution is using $N$-evaluation to aid the attack algorithm, which isn't burdensome because (1) increasing $N$ slightly raises costs, and (2) our algorithm remains stable without a large $N$. We use only the byproducts of $N$-evaluation, with the main overhead being the computation of approximate gradients, which is less costly than full gradients. Our algorithm's computational cost involves $N$ approximate gradients and one full gradient, whereas the EOT-based algorithm requires $N$ full gradients. Comparing computational times for varying $N$, we find that increasing $N$ doesn't significantly raise costs as shown in Rebuttal Figure.4. We also assess different $N$ values on algorithm performance, fixing evaluations at 10 for fairness. Results in Rebuttal Figure.5,6 show that a small $N$ (e.g. $N=5$) allows our algorithm to address the gradient dilemma and improve attack. Thus, $N$-evaluation effectively enhances attack efficiency at minimal cost.

Direct comparison with other papers

We computed robustness metrics using 1-ASR for direct comparison with other studies. Our differing results arise because the original paper reports the best historical robustness in 1-evaluation, while we focus on N-evaluations. Let $M$ denote iterations, and $n^{(N)}_m$ the successful attacks in the $m$-th iteration's N-evaluation. Our Avg.Rob is $1 - \max_m \sum n^{(N)}_m / N$, Wor.Rob is $1 - \max_m 1(n^{(N)}_m >0)$, while their Rob is $1 - \max_m 1(n^{(1)}_m >0)$. Their Rob is more akin to our Wor.Rob but from different samples, which can't assess model robustness for a given sample. After comparing methods consistently, most attack results align with the paper, except GDMP shows lower robustness due to the smaller WRN-28-10 classifier compared to DiffPure's WRN-70-16. To ensure fair benchmarking, we use WRN-70-16. Replacing it with WRN-28-10 reproduces GDMP results. Inconsistencies with LM results are due to numerical explosions in gradient computation which may disable the attack, we resolved it by gradient clipping.

Table 1: Comparison with other papers.

Other dataset

CIFAR10 is the primary dataset for evaluating adversarial defenses, so we focus on it. We also include results on other datasets, like restricted ImageNet and CIFAR100, where restricted ImageNet's 9 superclasses present more challenges. We reduced the attack budget to 4/255. Results show our algorithm's ASR significantly surpasses others, indicating the gradient dilemma's presence across datasets.

Table 2: Experiment result on restricted Imagenet

Table 3: Experiment result on CIFAR100

Thank you for the valuable feedback, here's our response to the concerned problems. We will describe our EM framework more clearly and rigorously in response to concerned problems, and we will also update this part of the exposition in our paper.

Given a sample $x\in\mathbb{R}^d$ and its label $y\in [K]$, a stochastic purification $\phi:\mathbb{R}^d\to \mathbb{R}^d$ and a deterministic classifier $f:\mathbb{R}^d\to \mathbb{R}^K$ will classify it as $f[\phi(x)]$. Denotes the normalized gradient $g(\phi):=\nabla_x \mathcal{L}(f[\phi(x)];y) /\lVert \nabla_x \mathcal{L}(f[\phi(x)];y) \rVert_2$, we suspect that $g(\phi)$ may obey a multi-peaked distribution, where $g(\phi)$ can be divided into clusters that have high intraclass similarity and low interclass similarity.

Our goal is to find the cluster center with the highest attack success rate. At each step of the attack, we can observe $N$ $g(\phi_i)$ and $\mathcal{A}_i, i=1,...,N$ (whether $f\circ \phi_i$ is attacked or not), We identify the parameters in the distribution of $g(\phi)$ by maximizing the likelihood function of $g(\phi_i)$ and $\mathcal{A}_i$.

We make the following assumptions about the distribution of $g(\phi)$: (1) There exists a master cluster of $g(\phi)$ that obeys a Gaussian distribution. Let the hidden variable $z$ denote whether $g(\phi)$ comes from this cluster, then $p(g(\phi)|z=1) = \mathcal{N}(r,\Sigma)$ and the prior distribution of $z$ is the Bernoulli distribution with parameter $\alpha$. The proportion $\alpha$, mean $r$, and variance $\Sigma$ of this cluster are the parameters of the underlying model that we desired. (2) We use a null information prior for whole $g(\phi)$, i.e., we assume that $p(g(\phi))=c$ and $c$ will be eliminated in subsequent analyses. We make these assumptions because the normal distribution is flexible in modeling similarity between $g(\phi)$, and the zero-information prior avoids over-designing our model. Furthermore, these assumptions facilitate subsequent theoretical derivations, and we will show how certain steps of the algorithm can be modified to accommodate other assumptions.

Since the hidden variable $z_i$ for each $g(\phi_i)$ is unknown, we use $q_i^{(t)}$ as an approximation of the $z_i$ posterior distribution at each stage $t$. Due to the properties of the Gaussian mixture model, the update involves only the mean of $q_i^{(t)}$ (i.e., the posterior mean of $z_i$: $\mathbb{E}[z_i|g(\phi_i)]=p(z_i=1|g(\phi_i))$), so in practice, $\mathbb{E}q_i^{(t)}$ can be viewed as an approximation of the posterior probability that $g(\phi_i)$ belongs to the main cluster. Then in M-step, $\mathbb{E}q_i^{(t)}$ is fixed and the parameters $r,\Sigma,\alpha$ are updated; in E-step, $\mathbb{E}q_i^{(t)}$ is updated to approximate $p(z_i=1|g(\phi_i))$.

M-step. In M-step, maximizing our objective (Eq. 6 in the paper) can be achieved by gradient ascent in Eq. 8 in the paper, where the observed $g(\phi)$ is weighted by $\mathbb{E}q_i^{(t)}$, and $r$ moves towards the weighted center.

E-step. As we discussed in Section 3.2.1 of the paper, we set $\Sigma$ as a hyperparameter, so we are only concerned with the update of $r$, where the previously assumed constant $c$ is eliminated.

where $A$ denotes (not a same) constants independent of $i$. In the last term of Eq. 8 in the paper, due to the $\alpha$ of the denominator, it will become $\sum_i[\exp(\cos\langle g(\phi_i), r^{(t)} \rangle)g(\phi_i)]/\sum_i\exp(\cos\langle g(\phi_i), r^{(t)} \rangle)$. We can find that this is a result weighted by a normalized similarity function w.r.t. $g(\phi_i), r^{(t)}$, and the function $\exp\cos\langle \cdot, \cdot \rangle$ is the result of our assumption.

Table1: DiffHammer in different assumption.

• Response to question 1: We will define $g(\phi)$ as the gradient and call a set with high intraclass similarity and low interclass similarity a cluster.
• Response to question 2 & 8: We adopt the assumption of Gaussian distribution for flexibility and convenience, which provides a theoretical reference for our algorithm. We experimented with other customized similarity functions in the above E-step, e.g. we directly defined the similarity of $g(\phi_i)$ to $r$ as the expectation of the loss boost (the closer of these indicates the stronger the attack). As shown in Table 1 below, proper similarity function choices leads to good results in practice, which in theory corresponds to similar but different cluster modeling.
• Response to question 3 & 4: Since $r,\Sigma$, and $\alpha$ are parameters of the underlying model, they are shared and unique among the $g(\phi)$.
• Response to question 5: $q_i^{(t)}$ is an approximation of the $z_i$ posterior distribution at each stage $t$ and only $\mathbb{E}q_i^{(t)}$ is involved and updated.
• Response to question 6: Maximizing the target with Eq. 6 is equivalent to making $r$ close to the $\mathbb{E}q_i^{(t)}$-weighted average $g(phi_i)$, which means that attacking a purifications with higher $\mathbb{E}q_i^{(t)}$.
• Response to question 7: In Eq. 8, $g(\phi_i)$ denotes the observed random variable and is therefore deterministic.
• Misclassification of models can constrain the deployment of models in safety-critical domains, for example, misclassification in autonomous driving can lead to catastrophic consequences. Diffusion-based sanitization serves as a promising solution, but may be undermined by resubmit attacks. Therefore, we reveal this risk to inspire subsequent research.

Thank you for your question. To define the cluster, we follow the definition in $1$ , where clusters are defined by "determine $m$ clusters (subsets) of individuals in $I$, and those individuals assigned to the same cluster are similar yet individuals from different clusters are different (not similar)".

Our assumption is that, for a class of purifications that can be attacked by the same adversarial noise, the gradients of purifications in this class follow a Gaussian distribution with mean $r$. We estimate $r$ as an approximation to the adversarial noise. We set a null-informative prior on other purifications that are attacked (but not by $r$) or that are not attacked. Thus, we have only a shared set of Gaussian distribution parameters.

In fact, it is not necessary to assume a Gaussian mixture model for all attacked purifications and assign $m$ sets of parameters ($r_1,.... .r_m;\Sigma_1,... ,\Sigma_m;\alpha_1,... ,\alpha_m$) where $\alpha_1>\alpha_2... ,>\alpha_m$. The reason for this is that the EM algorithm is locally convergent and we are only concerned with the cluster mean $r_1$ that has the largest proportion ($\alpha_1$). Purification from $\mathcal{N}(r_1,\Sigma_1)$ is more likely to occur in the first few batches of observations due to the larger probability $\alpha_1$, which allows the estimated $r$ in EM algorithm to approach $r_1$ initially and optimize towards $r_1$ thereafter. Although the EM algorithm also has a probability of converging to a suboptimal, this also has a sufficient attack effect and avoids the additional overhead of maintaining $m$ sets of parameters. Therefore, we only assume a set of Gaussian parameters $r,\Sigma,\alpha$ to make our algorithm efficient and concise.

$1$ Duran B S, Odell P L. Cluster analysis: a survey[M]. Springer Science & Business Media, 2013.

Thanks for the authors' efforts for further clarification! My concerns are now addressed and I raise the score to 6. I hope the authors can make the method part more clear and rigorous in revision.

Thank you for the valuable feedback, here's our response to the concerned problems.

Validation of the gradient dilemma

Summary: By conducting cluster analysis on the gradients of the purification process, we observed that the gradients have multiple clustering centers with low inter-cluster similarity, leading to what we term the "gradient dilemma" in attacks. This phenomenon is illustrated using a simple Gaussian mixture model, suggesting that the gradient dilemma may stem from non-clustered features in the data.

For our analysis, we sampled 500 gradients for each of the defenses. We employed a Gaussian mixture model (GMM) as the clustering algorithm, consistent with our paper's methodology. The optimal number of clusters is determined using the Akaike Information Criterion (AIC) within the range of 1 to 5. Our experiment revealed that 47% of the sampled gradients (across 3 defenses) possess more than one clustering center as shown in Rebuttal Figure 1. For these samples, we further analyzed the distribution of the cosine similarity between the top two cluster centers as shown in Rebuttal Figure 2, which was often in (-0.2 $\sim$ 0.3), thereby confirming the presence of the gradient dilemma.

To investigate the origins of the gradient dilemma, we constructed a Gaussian mixture toy example where the two diagonal components belong to the same category. In this scenario, diffusion-based purification pulls samples slightly towards the origin before diffusing them back into the data distribution. Given that the optimal classifier is a heteroscedastic classifier aligned with the coordinate axes, a sample's gradient direction is either horizontal or vertical, depending on which cluster the sample is closer to.

We demonstrated the probability of a sample's most frequent gradient direction in Rebuttal Figure 3, where values nearing 50% indicating the presence of a gradient dilemma. Simulation results revealed that samples along the diagonal experience severe gradient dilemmas, which is consistent with intuition. In this toy example, the gradient dilemma arises from a non-clustered data distribution, which imposes divergent gravitational pulls for purification. Consequently, we hypothesize that real-world data distributions may also exhibit non-clustered features. For instance, in the misclassification of a cat as a dog, considering the existence of various dog breeds, it is more efficient and effective to distort the features of one specific breed rather than all breeds collectively.

Analysis of DiffSmooth vulnerability

Summary: The robustness of DiffSmooth benefits from majority voting, a mechanism that enables certifiable robustness. However, its smoothed classifier remains vulnerable to DiffHammer. Although DiffSmooth is robust to resubmit attacks, this occurs at the expense of significant computational cost.

DiffSmooth employs a smoothing classifier in the inner loop and a certification algorithm in the outer loop. The inner loop's smoothing classifier can be viewed as an aggregation of multiple 1-step DiffPure processes. We evaluated DiffSmooth's performance under the $\ell_2$-norm using a ResNet-110 model trained with Gaussian smoothing, adhering to the paper's parameter settings. As illustrated in Figure 1, DiffSmooth's classifier alone is not sufficiently robust; DiffHammer achieves worst-case robustness of 57.0% and 86.7% (out of 10 evaluations) with perturbation budgets of 0.5 and 1, respectively.

Table 1: Experiment results on DiffSmooth classifier

The outer loop's certification algorithm derives robustness through majority voting, providing a lower bound on DiffSmooth's robustness and resisting resubmit adversarial samples with small attack success rate. Although majority voting suppresses resubmit attacks, it encounters several issues:

1. Achieving theoretical certification requires a vast number of samples (N = 100,000), limiting its practicality.
2. DiffSmooth can falsely certify a class when DiffHammer submits adversarial samples with higher ASR. Our experiments show that DiffSmooth will even falsely certify DiffHammer-generated adversarial samples with a radius of 0.175.

Thank you for the valuable feedback, here's our response to the concerned problems.

Comparison with newer methods

Summary: We adhere to the literature's naming conventions for these methods, but our evaluation incorporates the enhancements proposed in $1,2$ (2023), upgrading PGD and AA to SOTA evaluation with exact gradients. Additionally, we test complementary attacks.

Stochastic and iterative algorithms for diffusion-based purification yield challenging gradients, so a line of works aimed at approximating gradients in attack algorithms. This includes the Adjoint method $3$ and Joint methods (score/full) $4$ . $1,2$ obtained exact gradients of DiffPure and GDMP via computational graph reconstruction, and we further derive exact gradients for the optimization in LM using the Hessian vector product trick. It was shown that approximate gradients lead to inadequate robustness evaluation in $1,2$ . Therefore, we utilize exact gradient algorithms as a better baseline, with complementary experiments on approximate gradients validating this point. We will clarify our evaluation in the paper.

Transfer-based attack DTI

We include a demonstration and discussion of DTI, a representative transfer-based attack approach. We will compare DTI's performance with other algorithms in Figures 4 and 5 of the paper. DTI performs well in transfer-based attacks, likely due to its emphasis on translation and transformation (scaling and affine) invariance. Since most stochastic purifications do not share vulnerabilities, assuming global optimality can invalidate some transfer-based algorithms (VMI and CWA). Whereas DTI does not rely on this assumption, but it only considers simple transformations such as translation and scaling, which limits its performance.

Typos and presentation issues

• Since the time spent per step is comparable for the different methods, we directly report the number of iterations required to reach 90% of the optimal metric. We will unify the statements as iterations rather than time to avoid ambiguity.
• We define the robustness of the defense as the proportion of data that is not misclassified in 10 resubmit attacks, which is 1-Wor.ASR (the proportion of attacks that succeed at least once).
• We will add horizontal and vertical axis labels to the image to enhance legibility.
• We will check and correct typos in the paper.

$1$ Robust Evaluation of Diffusion-Based Adversarial Purification, ICCV, 2023.

$2$ DiffAttack: Evasion Attacks Against Diffusion-Based Adversarial Purification, NeurIPS, 2023.

$3$ Diffusion models for adversarial purification, ICML, 2022.

$4$ Adversarial purification with score-based generative models, ICML, 2021.

We sincerely appreciate your acknowledgment and your positive feedback on our work!