MAE-Pure: Semantic-Preserving Adversarial Purification

We sincerely thank the reviewer for the kind and constructive comments, which have greatly helped us improve the quality of our paper. We look forward to further engaging with you during the discussion phase.

W1: Extension experiment on CNN structure

We selected a CNN architecture similar to MAE, called Context Encoders [1], and proposed CE-Pure in a manner similar to MAE-Pure, to test on CIFAR-10 and SVHN, using WRN-28 as the classifier. Context Encoders are a CNN-based self-supervised learning method that learns contextual features of images by masking regions and reconstructing the missing content. Context Encoders and Masked Autoencoders (MAE) are both self-supervised learning methods based on masking. Their core idea is to mask part of the input and train the model to reconstruct the missing content, thereby learning meaningful feature representations. The main difference between the two lies in their architectural design: Context Encoders are based on CNNs, which are better at capturing local structural information and are commonly used for image inpainting tasks; in contrast, MAE is based on the Transformer architecture, which excels at modeling global contextual information. Context Encoders are a CNN-based self-supervised learning method that learns contextual features of images by masking regions and reconstructing the missing content. Alignment with main paper, we use PGD200+EOT20 with full gradient. The results are as follows:

Table: Comparison of CE-Pure and MAE-Pure results on CIFAR-10 and SVHN using WRN-28 as the classifier.

The results show that the robustness of MAE-Pure is significantly stronger than that of CE-Pure. This mechanism cannot be transferred to CNNs, thus proving that our theory is correct.

W2: Concerns about kernel coefficients

Thank you for your valuable feedback. Regarding the reconstruction bias constants (e.g., $C\_\text{rec}$) and the approximation of kernel coefficients (e.g., $\omega$ in Theorem 3.1), we agree that a deeper empirical exploration of these factors' sensitivity to the theoretical bounds is currently lacking. In future work, we plan to conduct more detailed empirical analysis to better understand the impact of these parameters on the theoretical results and optimize our estimation methods.

Additionally, concerning the notation and the placement of key definitions (e.g., B and Y in Theorem 3.1), we understand that this may affect the readability of the paper. In the revision, we will clarify the notation and ensure that key content is not placed in the appendix but is clearly explained in the main body of the text to improve understanding of the theoretical sections.

Once again, thank you for your suggestions. We will make sure to incorporate your feedback and further refine the relevant sections in the revision.

W3: Concerns about border impact

We thank the reviewer for highlighting the potential for misuse of our defense techniques in sensitive domains such as surveillance or biometric authentication. To address this concern concretely, we propose the following mitigation strategies:

1. Domain-Specific Safeguards: Our method (e.g., MAE-Pure, RMaskDiT-Pure) is not intended for direct use in surveillance or biometric applications without substantial adaptation. We recommend enforcing access control and fine-tuning with fairness-aware objectives when applied to such domains.

2. Auditability and Interpretability: We will release our codebase with tools for visualizing semantic attention maps before and after purification. This transparency enables third-party auditing and fosters responsible adoption.

3. Use-Case Licensing: We plan to adopt a license (e.g., OpenRAIL-style) that explicitly prohibits use in biometric identification, mass surveillance, or military applications.

4. Fairness and Bias Audits: While our semantic-preserving approach may improve robustness uniformly, we emphasize the need for demographic fairness audits before deployment in human-centric tasks to avoid unintended bias.

5. Collaborative Deployment Oversight: We advocate for collaborative oversight involving ethicists, legal experts, and application-domain practitioners when deploying purification models in safety-critical environments.

These strategies aim to ensure our method supports robustness research while minimizing the risk of unethical deployment. We also will add them in the revision late.

Q1: Extension to CNN-based autoencoders

Thank you for the question. As shown in our additional experiments addressing Weakness 1, MAE-Pure’s effectiveness is critically based on attention mechanisms. CNN-based autoencoders as context autoencoders, which lack attention matrices, show very week-round robustness. As a result, the purification does not generalize to CNN architectures. This confirms that attention is not optional, but essential for the design and success of our method.

Q2: manage trade-off We thank the reviewer for highlighting the slight drop in clean accuracy observed on CIFAR-100 when using RMaskDiT-Pure. This phenomenon stems from the trade-off introduced during the fine-tuning phase, where we incorporate a classification loss to enhance robustness. While this additional supervision effectively guides the model to focus on features invariant under adversarial perturbations, it may unintentionally suppress fine-grained features that are important for clean classification—especially on complex datasets like CIFAR-100, where inter-class distinctions are subtle.

To better manage this trade-off, we are exploring two main strategies. First, we aim to refine the hyperparameters involved in fine-tuning—such as adjusting the weight of the classification loss or employing early stopping based on clean validation accuracy—to prevent over-regularization. Second, we are investigating selective data filtering techniques that can reduce the influence of adversarially ambiguous or noisy samples. This can help the model retain more semantically aligned features during fine-tuning, thereby preserving clean accuracy while maintaining robustness.

Q3: Extension to other modalities

We appreciate the reviewer’s thoughtful question regarding the potential generalization of our AMV-based purification framework to non-image modalities such as audio and text.

While our current work is focused on the vision domain, we agree that the core idea of leveraging attention matrix variation (AMV) for detecting and correcting semantic distortions could be applicable to other modalities—especially those that rely on masked reconstruction mechanisms, such as masked language models in NLP (e.g., BERT) or spectrogram-based masked modeling in speech processing. These domains also involve attention-based architectures where inter-token or inter-frame semantic consistency plays a crucial role.

Exploring this cross-modal generalization is a promising direction, and we plan to investigate the extension of our approach to textual and audio data in future work. We believe such studies could further reveal the universality and flexibility of AMV-based semantic purification.

Q4: Releasing of visualizing tools

We thank the reviewer for the valuable suggestion. We fully agree that tools for visualizing the evolution of Attention Matrix Variation (AMV) during attacks and purification can be highly beneficial for the research community.

In our final release, we plan to open-source the complete codebase, including all components related to AMV computation and visualization. This will enable researchers to easily reproduce our results, analyze AMV dynamics under various settings, and potentially extend the visualization tools to other tasks and modalities.

[1] Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., & Efros, A. A. (2016). Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2536-2544).

We sincerely thank the reviewer for the kind and constructive comments, which have greatly helped us improve the quality of our paper. We look forward to further engaging with you during the discussion phase.

Weakness 1:Equivalence between the AMVM problem and the minimization of the MAE reconstruction loss

Thank you for your valuable question regarding the optimization equivalence relationship between attention matrix variant (AMV) and the reconstruction loss. We believe that the equivalence between the attention matrix variant minimization (AMVM) problem and the minimization of the MAE reconstruction loss holds. We have demonstrated in Theorem 3.1 and in Figure 4(a), (b), and (c) that adversarial perturbations lead to simultaneous increases in both the Attention Matrix Variation (AMV) and the MAE reconstruction loss. Moreover, as the magnitude of the adversarial perturbation increases, the trends of AMV and reconstruction loss consistently rise in a correlated manner. Furthermore, Theorem 3.2 in our paper already provides a rigorous inequality and can be transformed into:

$\frac{1}{2NT}\sum\_{t=1}^{T}\sum_{i=1}^{N} [HC\_A || \mathbf{A}^{\mathrm{dec}}\_{\mathrm{adv},i,t} -\mathbf{A}^{\mathrm{dec}}\_{i,t}||^2]\leq \mathcal{L}^{\mathrm{adv}}\_{\mathrm{rec}} - \frac{1}{2} \mathcal{L}\_{\mathrm{rec}}$

which demonstrates that the AMV objective is upper-bounded by $\mathcal{L}^{\mathrm{adv}}\_{\mathrm{rec}}$. Since $\mathcal{L}\_{\mathrm{rec}}$ remains constant for a well-trained MAE. As $\mathcal{L}^{\mathrm{adv}}\_{\mathrm{rec}}$ decreases, the AMV also gradually decreases.
Based on the above discussion, we therefore conclude that optimizing AMVM under the perturbation constraint \Delta_\{\infty} \leq C\_e is equivalent to minimizing the reconstruction loss, namely

$\min\_{\Delta}\mathcal{L}\_{\text{rec}}(x\_{\text{adv}}+\Delta)\Longleftrightarrow \min\_{\Delta}\mathcal{L}(\Delta) \qquad \text{s.t. } |\Delta|\_{\infty} \leq C\_e$

where $\min\_{\Delta}\mathcal{L}\_{\text{rec}}(x\_{\text{adv}} + \Delta)$ minimizes the MAE reconstruction loss for denoising, and $\min\_{\Delta} \mathcal{L}(\Delta) = \left\| \text{atten}(x\_{\text{adv}} + \Delta) - \text{atten}(x) \right\|\_2^2$ minimizes the attention matrix variation to preserve inter-patch semantic consistency. Theorem J.1 in the Appendix also ensures that optimizing Eq.(2) via PGD yields effective convergence, progressively guiding adversarial examples toward the semantic space of clean samples. This confirms the validity of Eq.(2) as a tractable and principled proxy for solving the original AMVM objective in Eq.(1). Moreover, the empirical analysis in Figure 4 (d) (e) and (f) further supports this theoretical derivation. The figure 4 (d),(e) and (f) shows that as we optimize $\min\_{\Delta}\mathcal{L}\_{\text{rec}}(x\_{\text{adv}}+\Delta)$, the reconstruction loss decreases, and the AMV decreases accordingly.

Therefore, we believe our explanation of the equivalence between AMVM and the minimization of the MAE reconstruction loss is both sufficient and well-justified.

W2: Concerns about visualization

Thank you very much for this valuable comment. We appreciate your careful reading of our paper.

Regarding the concern that “the visual differences in the attention weights between the clean, adversarial, and denoised examples are quite subtle”, we would like to clarify that these differences are actually very evident in Figure 2. For example, in the clean image shown in Figure 2(b), the target patch (highlighted by the red box) corresponds to a hole on the magnetic tape, and its most relevant image patch is another hole on the tape---this reflects a clear semantic relation. However, in the adversarial image shown in Figure~2(c), the same target patch’s most relevant patch is shifted to an unrelated edge region. This demonstrates that the MAE’s attention has been significantly distorted by the adversarial perturbation, which we believe makes the visualization itself quite convincing.

In addition to this qualitative visualization, we also provide quantitative evidence: as shown in Figure~2(a), (b), and (c), we report that increasing the magnitude of adversarial perturbation leads to a measurable increase in the attention matrix variation (AMV). This directly supports our claim that adversarial perturbations disrupt semantic relations as reflected in the attention mechanism.

We hope this explanation clarifies why we believe the presented visualization, together with the quantitative AMV analysis, strongly supports our claim.

W3: Limited attack evaluation

Thank you very much for this insightful comment. We sincerely appreciate your careful review and constructive suggestions.

Regarding the concern that our methodology and experiments might focus on a limited set of attacks, we would like to clarify that our evaluation already covers a relatively broad range of adversarial attacks. Specifically, in Table 1, Table 2, and Table 4, we present results under PGD200+EOT20. In Table 3, we additionally report results under AutoAttack. Moreover, in Table 7, we further include evaluations with CW+EOT and DeepFool+EOT, and in Table 11, we provide results under BPDA.

These experiments together demonstrate the robustness of our method under diverse attack strategies, not limited to AutoAttack. We hope this clarifies that our defense is not tailored to a single type of attack but has been validated across multiple strong and widely recognized adversarial attack methods.

W4: Typos correctness

Thank you very much for your kind correction of our typos. We will address and fix these errors in the revision.

Q1：Clarification on MAE-Pure as a Counter-Adversarial Process

Yes, your understanding is correct. The MAE-Pure process can indeed be considered a form of "counter-adversarial" transformation applied to adversarial examples. Its objective is to reduce the reconstruction loss (i.e., MAE loss), thereby suppressing the variation introduced in the AMV. By guiding the reconstruction towards the clean manifold, the purifier effectively neutralizes adversarial perturbations and stabilizes the downstream representation.

Q2: Clarifying and Emphasizing Our Contributions in the Revision

Thank you for your valuable feedback. We sincerely appreciate your suggestion and will revise the writing to better highlight the key contributions of our work.

Dear Reviewer DP9v,

I hope this message finds you well. As the discussion period is nearing its end with less than three days remaining, we would like to ensure that all your concerns have been addressed satisfactorily. If there are any additional points or feedback you would like us to consider, please don’t hesitate to let us know. Your insights are invaluable to us, and we are eager to resolve any remaining issues to further improve our work.

Thank you again for your time and effort in reviewing our paper.

Best

Regard

Team 21073

Dear Reviewer DP9v

We sincerely appreciate your response and are pleased that we were able to address some of your concerns. Moving forward, we hope that further discussion will help clarify any remaining questions you may have.

For Equivalence

We would like to clarify that our formulation is mathematically equivalent in the strict sense. As you rightly noted, AMV is upper-bounded by $\mathcal{L}^{\text{adv}}\_{\text{rec}}$, which implies that as $\mathcal{L}^{\text{adv}}\_{\text{rec}} - \frac{1}{2} \mathcal{L}\_{\text{rec}}\rightarrow 0$, AMV will also converge to zero, because AMV is non-negative.

Furthermore, we would like to clarify that minimizing an upper bound to indirectly optimize an intractable or non-differentiable objective is a well-established and widely accepted practice in the machine learning and optimization community [1,2,3,4]. We thus hope this clarification helps reconcile the concern regarding the optimization objective.

For the Sake of Rigor

We would like to clarify that the approximations used in our theoretical analysis, particularly the kernel approximation and the Taylor expansion, are not arbitrary choices. Rather, they are well-established and widely adopted techniques in the machine learning and optimization communities. These methods have been extensively employed in numerous prior works, especially in contexts such as [5,6,7,8], and have been repeatedly validated in influential papers published in top-tier journals and conferences. We emphasize that our use of these techniques follows standard practices in the field and builds upon the stability and guarantees established in the existing literature. Therefore, we are confident that our analysis adheres to the rigorous standards upheld by prior works in top-tier venues, and we welcome any suggestions for further clarification or strengthening.

For Visualization

We present three sets of examples in our paper: the first is shown in Figure 2, while the second and third are included in Figure 8 of the appendix. In particular, the second row of Figure 8 clearly illustrates how the adversarial noise affects the semantic content. This example was carefully chosen to make the semantic impact of the perturbation visually salient and intuitive.

We sincerely appreciate your response once again. Should there be any further questions, we would be happy to engage in continued discussion.

Best Regards

Your Friends

Team 21073

[1] Hinton, G., Vinyals, O., & Dean, J. (2015). Distilling the knowledge in a neural network. NIPS 2017

[2] Wong, E., & Kolter, Z. (2018, July). Provable defenses against adversarial examples via the convex outer adversarial polytope. In International conference on machine learning (pp. 5286-5295). PMLR.

[3] Bartlett, P. L., Jordan, M. I., & McAuliffe, J. D. (2006). Convexity, classification, and risk bounds. Journal of the American Statistical Association, 101(473), 138-156.

[4] Zhang, S., Qian, Z., Huang, K., Wang, Q., Zhang, R., & Yi, X. (2021, July). Towards better robust generalization with shift consistency regularization. In International conference on machine learning (pp. 12524-12534). PMLR.

[5] Moosavi-Dezfooli, S. M., Fawzi, A., & Frossard, P. (2016). Deepfool: a simple and accurate method to fool deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2574-2582).

[6] Neyshabur, B., Bhojanapalli, S., McAllester, D., & Srebro, N. (2017). Exploring generalization in deep learning. Advances in neural information processing systems, 30.

[7] Choromanski, K., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., ... & Weller, A. (2020). Rethinking attention with performers. NeurIPS 2021.

[8] Nguyen, T., Pham, M., Nguyen, T., Nguyen, K., Osher, S., & Ho, N. (2022). Fourierformer: Transformer meets generalized fourier integral theorem. Advances in Neural Information Processing Systems, 35, 29319-29335.

Dear reviewer DP9v

Thank you very much for your kind and constructive review.

Regarding your previous concerns on Equivalence Justification, Theoretical Rigor, and Visualization Clarity, we believe we have now addressed them thoroughly in our discussion.

For Equivalence Justification and Theoretical Rigor, our formulation strictly follows the methodology adopted in several influential prior works as shown in last discussion. These references provide the theoretical foundation for our approach, and we have ensured that our derivation remains faithful to the rigor established in these studies.

For Visualization Clarity, we have shown more illustrative examples in Figure 8, which we hope make the semantic effects of adversarial perturbations more intuitively observable.

Please let us know if there are any remaining or new concerns—we would be more than happy to clarify or further elaborate.

Warm regards,

Your Friends,

Team 21073

We sincerely thank the reviewer for the kind and constructive comments, which have greatly helped us improve the quality of our paper. We look forward to further engaging with you during the discussion phase.

W1 and W2

Thank you very much for your valuable comment, and we sincerely apologize for not having made our motivation clear in the original manuscript.

Our motivation for choosing the MAE architecture is based on its observed sensitivity to adversarial perturbations when using attention mechanism for reconstruction, as shown in Figure 1. We further investigated the root cause and found that this sensitivity originates from the attention matrix in MAE, which is highly susceptible to noise. This is theoretically justified in Theorem 3.1 and empirically validated in Figure 3 (a), (b), and (c).

Based on this phenomenon, we propose MAE-Pure, which aims to suppress adversarial noise by constraining the variation of the attention matrix (AMV). This mechanism is grounded in the proven consistency between AMV and reconstruction loss, as established in Theorem 3.2 and further supported by Figure 3 (d), (e), and (f). Therefore, by optimizing the reconstruction loss, MAE-Pure effectively reduces AMV, thus achieving denoising.

Furthermore, in Figure 7, we demonstrate that the same characteristics hold for masked diffusion-based model, e.g., MaskDiT. Leveraging these insights, we integrated our method into MaskDiT and observed consistent and strong performance.

Q1: Selection for MAE

Thank you for your valuable comment. While diffusion-based models have recently demonstrated strong performance in purification tasks, our motivation for using MAE based on two reasons: (1) To identify the purification effectiveness is solely produced by minimizing Attention Matrix Variation (AMV), we choose the MAE which only includes the attention mechanism. (2) To generalize our observation to a more complex model architecture, we integrate our AMV minimization into MaskDiT that not only includes the attention mechanism but also embodies powerful generation capability.

In our paper, we provide both empirical and theoretical evidence for the effectiveness of MAE as a purifier. Specifically, in Figure 4(a), (b), and (c) and Theorem 3.1, we show that adversarial perturbations lead to significant variations in the attention matrices of MAE. These variations are systematically aligned with the adversarial noise, revealing an intrinsic vulnerability.

Building upon this, we introduce the concept of AMV as a quantitative measure of perturbation sensitivity. Furthermore, in Figure 4(d), (e), and (f) and Theorem 3.2, we demonstrate that minimizing AMV provides a principled and effective strategy for denoising adversarial noises.

More importantly, our insights are not limited to MAE. As shown in Figure 7, the AMV-based purification principle can be seamlessly extended to MaskDiT leading to our proposed method MaskDiT-Pure. By leveraging the strong generative capability of diffusion models, MaskDiT-Pure achieves superior performance compared to existing other diffusion-based purification baselines.

In summary, our use of MAE is not only empirically validated but also theoretically motivated, and the proposed framework generalizes beyond MAE to improve even diffusion-based purifiers.

Q2: Concerns about runtime

Thank you for your question. The observed runtime difference stems from the fact that DiffPure employs different diffusion models on CIFAR-10 and ImageNet. Specifically, for CIFAR-10, DiffPure uses a score-based SDE model, while for ImageNet, it adopts a Guided Diffusion.

Notably, guided diffusion is known for its stronger generative capacity and better sample quality, but it also incurs higher computational overhead compared with score-based diffusion. As a result, even excluding the influence of input size, DiffPure takes significantly more time on ImageNet than on CIFAR-10, primarily due to the heavier computational cost of the guided diffusion process.

To summarize, the difference in runtime patterns mainly arises from DiffPure's use of different diffusion models, whereas our method ensures consistency by design. This also explains why our method shows the higher runtime on CIFAR-10 but the lower runtime on ImageNet compared to DiffPure.

Q3: Comparison with more different baselines

Thank you for the constructive suggestion. Following your recommendation, we have included a comparison with two recent works that adopt Masked Autoencoders (MAE) as purifiers: MAEDefense [1] and MAEP [2]. % We reproduced their methods on both CIFAR-10 and SVHN datasets under a strong adaptive attack setting using PGD-200 + EOT-20 with full gradient backpropagation, which aligns with the evaluation protocol used in our paper. The backbone classifier for all methods is WideResNet-28-10 for a fair comparison.

The results are summarized below:

Table: Comparison with new MAE baseline on CIFAR10 and SVHN

We will add the results in our revision.

Q4: Optimaztion for upper bound

Thank you for your insightful comment. % We agree that minimizing an upper bound can effectively reduce the target loss, such as $\mathcal{L}\_{\text{rec}}^{\text{adv}}$. However, our method does not involve constructing or optimizing any upper bound of $\mathcal{L}\_{\text{rec}}^{\text{adv}}$; instead, we directly minimize $\mathcal{L}\_{\text{rec}}^{\text{adv}}$ itself by iterative update the adversarial examples for purification. Therefore, finding an upper bound does not provide direct benefit to our current approach.

Moreover, please see the reformulated inequality of Theorem 3.2 in response for Reviewer DP9v. We would like to clarify that the purpose of Theorem~3.2 is not to establish a lower bound of $\mathcal{L}\_{\text{rec}}^{\text{adv}}$, but rather to show that the attention matrix variation (AMV) term, i.e., $\sum\_{t=1}^{T}\sum\_{i=1}^{N}\left\| \mathbf{A}^{\mathrm{dec}}\_{\mathrm{adv},i,t} - \mathbf{A}^{\mathrm{dec}}\_{i,t} \right\|^2,$ can be upper-bounded by $\mathcal{L}\_{\text{rec}}^{\text{adv}}$.

This holds because the reconstruction loss of clean samples remains approximately constant for a well-trained MAE. As a result, minimizing $\mathcal{L}\_{\text{rec}}^{\text{adv}}$ implicitly reduces AMV, which helps preserve the inter-patch semantic relationships and thus achieves denoising.

Q5: Exploration of Figure 3

Thank you for the suggestion. We have added detailed explanations to Figure 3 to clarify the denoising pipeline of MAE-Pure.

Specifically, Figure 3 provides an overview of the proposed purification framework. It illustrates the iterative denoising process applied to adversarial examples. Starting from an adversarial input $x^{\text{adv}}$, the model applies Projected Gradient Descent (PGD) iterations, where at each step $s$, the reconstruction loss of the MAE is minimized to generate a purified sample $x^{\text{adv}}\_s$. This iterative process aims to reduce the Attention Matrix Variation (AMV) by aligning the attention distributions of adversarial and clean inputs. The figure also depicts the relationship between the MAE reconstruction loss and the adversarial perturbations being gradually removed, ultimately yielding a semantically faithful, purified image. The shaded areas and clip operations represent the bounded perturbation constraints ($\| \Delta \|\_{\infty} \leq C\_e$) used during optimization.

We will include the improved explanation in the revision.

Q6: Introduction of Large MAE

Thank you for your helpful comment. The term Large-MAE refers to the ViT-Large architecture introduced in the original MAE paper [3]. We will add the appropriate citation and clarify this description in Section~L of the supplementary material in the revision to improve reproducibility and reader understanding.

Q7: Coding problem

Thank you for your kind suggestion. We will address this issue in the revision.

[1] Lyu, W., Wu, M., Yin, Z., & Luo, B. (2023, October). Maedefense: An effective masked autoencoder defense against adversarial attacks. In 2023 Asia Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) (pp. 1915-1922). IEEE.

[2] Chen, Y. C., & Lu, C. S. (2025). Adversarial Masked Autoencoder Purifier with Defense Transferability. arXiv preprint arXiv:2501.16904.

[3] He, K., Chen, X., Xie, S., Li, Y., Dollár, P., & Girshick, R. (2022). Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 16000-16009).

Dear Reviewer tuPn

We sincerely appreciate your positive and friendly response, and we are also glad to have resolved some of your concerns. For your remaining questions, we are more than willing to engage in further communication to provide clarification.

For Q1: We thank the reviewer for the helpful clarification. We understand that the concern lies in the core problem encountered when using diffusion as a purifier, rather than the motivation for using MAE per se. Existing diffusion-based methods, such as DiffPure aim to project adversarial examples back toward the clean data distribution. However, they tend to overlook the analysis and optimization of semantic dependencies between image patches (i.e., inter-patch relationships), which are critical for effective denoising. Through our empirical analysis, we observe that inter-patch relationships are particularly sensitive to adversarial noise. This suggests, from an intuitive perspective, that explicitly modeling and preserving these dependencies could lead to better purification outcomes. Importantly, our proposed mechanism is not in conflict with diffusion-based approaches. Rather, it represents a general principle that can be instantiated in different architectures, including autoencoder-based models (e.g., MAE) and diffusion-based models (e.g., MaskDiT). Our results demonstrate that incorporating this principle consistently enhances denoising performance across both types.

For Q2: We agree with the reviewer that MAE is generally a lighter model compared to diffusion-based models. However, in our setting, we apply the MAE-based denoising process iteratively over multiple steps to achieve strong purification performance. This iterative process significantly increases the overall runtime, which explains the relatively high time cost despite the lightweight nature of the MAE architecture.

For Q4: Thank you for your insightful comment. To clarify, our objective is to minimize the term $\sum_{t=1}^{T}\sum\_{i=1}^{N}\left\| \mathbf{A}^{\mathrm{dec}}\_{\mathrm{adv},i,t} - \mathbf{A}^{\mathrm{dec}}\_{i,t} \right\|^2,$ which measures the distortion in the decoded attention maps under adversarial perturbation. We achieve this by monitoring and optimizing its upper bound, specifically $\mathcal{L}^{\mathrm{adv}}\_{\mathrm{rec}} - \tfrac{1}{2} \mathcal{L}\_{\mathrm{rec}},$ where $\mathcal{L}\_{\mathrm{rec}}$ is treated as a constant for a well-trained MAE. This approach allows us to indirectly constrain the adversarial distortion via an analytically tractable surrogate.

In addition, we also agree with your statement regarding obtaining an upper bound of the loss $\mathcal{L}^{\mathrm{adv}}\_{\mathrm{rec}}$. Therefore, we derive the loss upper bound as follows. $\mathcal{L}^{\mathrm{adv}}\_{\mathrm{rec}} \leq \mathcal{L}\_{\mathrm{rec}} + \frac{1}{(n-m)}\left| (L^2+1)||\delta||^2 \right|$, where $\mathcal{L}$ is the Lipschitz constant of the MAE. $N$ represents the total number of samples, $n$ denotes the number of patches per sample, $m$ refers to the number of masked patches, and $\delta$ represents the perturbation. The inequality shows that, under the given MAE architecture and data, the adversarial reconstruction loss $\mathcal{L}^{\mathrm{adv}}\_{\mathrm{rec}}$ is determined by the perturbation $\delta$. Since $\delta$ is typically small, this implies that the upper bound of $\mathcal{L}^{\mathrm{adv}}\_{\mathrm{rec}}$ remains controllable.

The proof is as follows:

Let $x\_{i,1}$ denote the unmasked (visible) part of the $i$-th sample, and $x\_{i,2}$ denote the masked (invisible) part of the $i$-th sample, consistent with the notation used in the main paper.

$\mathcal{L}^{\mathrm{adv}}\_{\mathrm{rec}} = \frac{1}{N(n-m)}\sum_{i=1}^{N}||g(f(x\_{i,1}^{adv}))-x\_{i,2}^{adv}||^2$

$= \frac{1}{N(n-m)}\sum_{i=1}^{N}\left|||g(f(x\_{i,1}^{adv}))-x\_{i,2}^{adv} + g(f(x\_{i,1}))+x_{i,2} - g(f(x\_{i,1}))-x\_{i,2} ||^2\right|$

$\leq \frac{1}{N(n-m)}\sum_{i=1}^{N}\left|||g(f(x\_{i,1}^{adv}))- g(f(x\_{i,1})) ||^2 + ||g(f(x\_{i,1})) - x\_{i,2} ||^2 + ||x\_{i,2}^{adv} - x\_{i,2}||^2\right|$

$\leq\frac{1}{N(n-m)}\sum\_{i=1}^{N}||g(f(x\_{i,1})) - x\_{i,2} ||^2+ \frac{1}{N(n-m)}\sum_{i=1}^{N}\left| ||x\_{i,2}^{adv}-x\_{i,2}||^2 + ||g(f(x\_{i,1}^{adv})) - g(f(x\_{i,1}))||^2\right|$

Based on a widely used assumption that $||g(f(a))- g(f(b))|| \leq L||a-b||$, we can get

$\leq\mathcal{L}\_{\mathrm{rec}}+ \frac{1}{N(n-m)}\sum\_{i=1}^{N}\left| ||x\_{i,2}^{adv}-x\_{i,2}||^2 + L^2||\delta||^2\right|$

$\leq \mathcal{L}_{\mathrm{rec}} + \frac{1}{(n-m)}\left| (L^2+1)||\delta||^2 \right|$

We sincerely thank the reviewer for the professional and constructive comments. These suggestions have greatly improved the quality of our paper. We will incorporate all the points you mentioned into our revision.

Best Regards,

Your friends,

Team 21073

Dear Reviewer tuPn,

I hope this message finds you well. As the discussion period is nearing its end with less than three days remaining, we would like to ensure that all your concerns have been addressed satisfactorily. If there are any additional points or feedback you would like us to consider, please don’t hesitate to let us know. Your insights are invaluable to us, and we are eager to resolve any remaining issues to further improve our work.

Thank you again for your time and effort in reviewing our paper.

Best

Regard

Team 21073

We sincerely thank the reviewer for the kind and constructive comments, which have greatly helped us improve the quality of our paper. We look forward to further engaging with you during the discussion phase.

Weakness 1: Standard deviations

Thank you for the reviewer’s kind suggestion. We have included the standard deviations for CIFAR-10, CIFAR-100 here SVHN. The results are obtained from 5 random runs.

Table: Results on CIFAR10.

Table: Results on CIFAR100 and SVHN (left side is CIFAR100 and right side is SVHN).

Weakness 2: Missing citation

Thank you for pointing this out. We will address the missing citation in line 93 and ensure it is properly included in the revision.

Weakness 3 :Concerns or clarifications regarding the threat model

We appreciate the reviewer’s insightful comment. In our experiments, the threat model is indeed the purification model combined with a classifier (e.g. MaskDiT + WRN28). This setting is consistent with prior works [1,2], and we adopted the same configuration to ensure a fair and direct comparison. % We will make a more clear exploration in the revision.

W4: Cost of different method

We presented the training time of the MAE-Pure/MaskDiT-Pure purifier and compared it with the DiffPure purifier. At the same time, we also performed a finetuning time comparison between RMAE-Pure/RMaskDiT-Pure and ADDT. We used 8* A40 GPUs for parallel training with a batch size of 128.

Table: Time consumption for model training (in hours).

Next table is finetuning time.

Table: Time consumption for robust model fine-tuning (in hours).

Q1: Ablation on the Effectiveness of MaskDiT framework

Thank you for your thoughtful question about whether the performance gain comes from the proposed Pure framework or the backbone architecture itself. We have already conducted ablation studies to address this concern in the submitted Supplementary Material. Please note that this part is not included in the Appendix. We present the key results in Supplementary Material below for clarity:

Table: Ablation analysis on different denoising components across various datasets

We begin with training two models, e.g., MaskDiT and RMaskDiT, used for purification. Firstly, following the DiffPure [3], we conduct the reconstruction-based purification processes for purification, which are MaskDiT$_{\text{/purification}}$ and RMaskDiT$\_{\text{/purification}}$. This reconstruction based purification involves a forward (noise addition) and backward (denoising) pass. The reconstructed images are treated as purified results to investigate whether the superior performance stems from generative ability of selected models. On the other hand, we then compared the above performances with the proposed AMV-based purification methods by using the same models, where the purification processes are named as MaskDiT-Pure and RMaskDiT-Pure. By keeping the model architecture consistent across all variants, we enabled a direct comparison of the effectiveness of different denoising strategies. It is note that MaskDiT and RMaskDiT indeed possess a certain capability to withstand stronger adversarial attacks; however, their effectiveness is still much lower than that of our proposed MaskDiT-Pure and RMaskDiT-Pure.

Q2: Motivation experiments on MaskDiT

We assume that by "motivation experiments" you are referring to Figure 2, which illustrates the impact of adversarial perturbations on the attention matrix in MAE. We believe that the motivations and observations from these experiments are consistent and applicable to MaskDiT.

To support the above conclusion, we have conducted the quantitative analysis about the relationship among the adversarial perturbations, reconstruction loss, and attention matrix variance (AMV) in Figure 7. The results demonstrate that MaskDiT exhibits a similar pattern with MAE, where adversarial perturbations lead to noticeable changes in attention variance and increased reconstruction loss. This consistency suggests that the motivation, namely that adversarial noise disrupts attention-based semantic representations, also applies to MaskDiT. Therefore, we conclude that the motivation remains valid for MaskDiT based on our empirical evidence.

Q3: Justification for Dataset Selection in Motivation Experiments

We appreciate the reviewer’s thoughtful question. The datasets were intentionally chosen to align with the specific phenomena we aimed to highlight in different parts of our empirical analysis. In Figure 1, SVHN was selected to study the impact of adversarial perturbations on MAE’s reconstruction quality. SVHN images are relatively simple and contain less semantic variation, making the degradation caused by adversarial noise more visually apparent and interpretable. This helps clearly demonstrate reconstruction failure under perturbations. In contrast, Figure 2 focuses on analyzing how adversarial perturbations disrupt the semantic relationships between image patches. For this purpose, we used ImageNet, as its images contain richer semantics, diverse objects, and complex spatial structures. This allows us to more convincingly showcase the disruption in inter-patch attention and the resulting semantic inconsistency.

Q4: ViT-based method

Thank you for the reviewer’s question and suggestion. Indeed, in our current experiments, we have only used ResNet-based architectures to validate the effectiveness of the robust purification model. To address the reviewer's suggestion regarding the use of ViT-based architectures (e.g., ViT-B), we plan to conduct further experiments on the CIFAR-10 dataset and use the ViT-B (Vision Transformer) model as the downstream classifier to evaluate the performance of our proposed method under this architecture. We use PGD200+EOT20 with full gradient aligned with main paper.

Table: Clean and robust accuracy (%) on CIFAR-10 obtained by different purification methods, where ViT-B is considered

The results show that the performance is worse compared to WRN-28. Our analysis suggests that this may be due to the fact that the ViT architecture is a data-hungry model, requiring larger datasets to fully leverage its potential.

[1] Adbm: Adversarial diffusion bridge model for reliable adversarial purification.

[2] Towards Understanding the Robustness of Diffusion-Based Purification: A Stochastic Perspective

[3] Diffusion models for adversarial purification.

Dear Reviewer eQ1q,

I hope this message finds you well. As the discussion period is nearing its end with less than three days remaining, we would like to ensure that all your concerns have been addressed satisfactorily. If there are any additional points or feedback you would like us to consider, please don’t hesitate to let us know. Your insights are invaluable to us, and we are eager to resolve any remaining issues to further improve our work.

Thank you again for your time and effort in reviewing our paper.

Best

Regard

Team 21073

Dear Reviewer eQ1q,

Thank you very much for your meaningful and constructive suggestions. We truly appreciate the time and thought you have devoted to reviewing our work.

We would like to let you know that we have carefully addressed your concerns regarding the experimental setup. Specifically, we have added the missing standard deviations and clarified the runtime cost of different purification pipelines.

In addition, we have included new ablation study to isolate the contribution of the proposed MaskDiT-Pure designs, and have also added a ViT-based classifier in our evaluation to enhance the generality of our findings.

We hope these clarifications and additional experiments adequately address your concerns. Please don’t hesitate to let us know if you have any further comments or recommendations. We would greatly appreciate the opportunity to further discuss with you.

Best regards,

Your Friends

Team 21073

Dear Reviewer eQ1q,

Thank you very much for your response and for taking the time to reconsider our work. We truly appreciate your thoughtful feedback and are grateful that our rebuttal was able to address your concerns. We're also sincerely thankful for your updated score and support.

Best regards,

Your Friends

Team 21073