MeanSparse: Post-Training Robustness Enhancement Through Mean-Centered Feature Sparsification

Thank you for your detailed review and valuable feedback. Below, we address your concerns:

$**Generalization Across Attention-Based Models:**$

Our primary objective in this paper is to compare the performance of MeanSparse-integrated models with the original adversarially trained ones. While there are several robust CNN architectures, the options for attention-based architectures are limited. Additionally, training well-robustified models is time-consuming. In the current version of the paper, we include the Swin-L transformer from attention-based architectures and ConvNeXt-L. Although ConvNeXt-L is not attention-based, it draws inspiration from the design principles of vision transformers. Our results demonstrate that MeanSparse is effective for both architectures.

$**Smaller Models:**$

We have conducted experiments to evaluate the effectiveness of MEANSPARSE on smaller models, such as ResNet-18, as part of our ablation studies. These experiments examined several key factors that could influence robustness. Across various settings, we demonstrated that MEANSPARSE improves robustness on smaller models as well. For a detailed analysis of the ResNet-18 results, please refer to Appendix A. Furthermore, Figure 3 presents results for models with varying parameter sizes and architectures, highlighting the effectiveness of our method across diverse model types.

$**Statistics During Test:**$

The running mean and variance are fixed during testing, ensuring that no parameters are changed during testing, and are computed using the training set as it provides a more reliable estimation of the mean and variance across different features.

$**Other Sparsity Methods:**$

MeanSparse is a post-training operator applied to feature vectors, which sets it apart from traditional sparsity methods typically used for regularization. While sparsity metrics such as the L1 norm [1] and SL0 regularization [2] could be alternatives, their proximal operators (or shrinkage functions) impact all feature values. In theory, every feature is altered based on its distance to the mean, leading to a significant decline in both clean and robust accuracy, making them unsuitable for post-training adjustments. In contrast, the hard-thresholding operator used in MeanSparse only affects features near the mean, preserving distant features that carry valuable information. This selective impact makes the L0 sparsity metric more effective in this context. Furthermore, to our knowledge, no existing research has demonstrated the use of sparsity shrinkage to enhance model robustness.

[1] Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological), 58(1), 267–288. https://doi.org/10.1111/j.2517-6161.1996.tb02080.x

[2] Mohimani, H., Babaie-Zadeh, M., Gorodnitsky, I., & Jutten, C. (2010). Sparse Recovery using Smoothed $\ell^ 0$(SL0): Convergence Analysis. arXiv preprint arXiv:1001.5073.

$**Questions:**$

$**1-**$ Accessing high-probability feature regions is crucial for MeanSparse. During training, as features are being learned, their statistics continuously change, making them unsuitable for our purpose. To address this, we compute the required statistics in a single additional epoch after training.

The paper also discusses integrating MeanSparse during training, which requires a carefully designed threshold scheduler. If the threshold increases too quickly, it disrupts training by causing the gradient zeroing effect of MeanSparse. Conversely, if it increases too slowly, the behavior resembles post-training integration. Furthermore, aligning the threshold scheduler with the activation functions is critical. These complexities make it challenging to incorporate MeanSparse into large models during the initial training phase. We include a detailed comparison of MeanSparse integration during and post-training in Section 3.3 of the revised manuscript.

$**2-**$ During our ablation study, we analyzed the effectiveness of MEANSPARSE across different attack strengths. Specifically, we measured the APGD accuracy of the ResNet-18 model with the GELU activation function on the CIFAR-10 test set before and after integrating MEANSPARSE under varying attack powers. For instance, with a threshold of 0.2, we observed that the robust accuracy was slightly lower than the base model (threshold = 0) for attack powers of ϵ=1/255 and ϵ=2/255. This decrease is primarily attributed to the reduction in clean accuracy caused by applying MEANSPARSE. However, for attack powers ϵ>2/255, MEANSPARSE consistently improved the APGD accuracy, demonstrating its effectiveness against stronger attacks. For example, at ϵ=16/255, APGD accuracy improves from 11.97% (base model) to 13.41% (threshold 0.2) and 15.44% (threshold 0.35).

Please refer to Appendix A.8 and Table 9, which provide a comprehensive analysis across different attack strengths.

Dear Reviewer,

Thank you for your thoughtful review and for highlighting both the strengths and areas for improvement in our work. Below, we address your concerns:

$**Ambiguity Regarding "Features":**$

The MeanSparse operator is typically placed between the batch normalization layer and the activation function. By "feature," we refer to the output of the layer preceding the MeanSparse operator, which is usually the output of the batch normalization layer.

$**Feature Mean as a High-Probability Point:**$

Precisely characterizing the mode of the distribution for a specific feature in a deep architecture is a complex task due to both the input distribution and the intricate mapping within the network. However, previous studies have demonstrated that feature distributions, particularly in the deeper layers of neural networks, tend to be unimodal [1], allowing the mean value to act as a representative of the high-probability region. Consequently, variations around the mean carry less significant information about the output. Additionally, we observe that after sparsifying the features around the mean, there is almost no reduction in accuracy, further validating our assumption that the mean is a reliable representative of the high-probability region.

$**Threat Models Beyond$\ell_{\infty**$and$\ell_{2}:}

Thank you for your insightful feedback. Our method is not restricted to $\ell_{\infty}$-bounded attacks; its effectiveness has also been demonstrated on state-of-the-art models robust to $\ell_{2}$-bounded attacks. For example, we applied MeanSparse to the WideResNet-70-16 model, which is state-of-the-art on CIFAR-10 with an $\ell_{2}$-attack budget and a radius of 0.5. By integrating MeanSparse, the clean accuracy remained nearly unchanged (95.54% to 95.49%), while robustness significantly improved (84.97% to 87.28%), highlighting its effectiveness against $\ell_{2}$-bounded attacks.

You can refer to Section 4.2 and Figure 3 for results demonstrating MeanSparse's robustness under $\ell_{2}$-bounded threats. Additionally, Appendix A.2 provides a detailed analysis showing how our method enhances robustness against both $\ell_{2}$- and $\ell_{\infty}$-bounded attacks. However, for the majority of our ablation studies, we adopt an $\ell_{\infty}$-bounded threat model, as it is the most commonly reported in the research community. We will include a note in the limitations section of the revised version to explicitly state that the attacks considered in this work are bounded, addressing your concern.

$**Questions:**$

$**1-**$ As discussed earlier regarding weaknesses, accurately tracking the exact high-probability region is complex because it requires estimating densities using tools such as feature histograms. However, the distributions in the deeper layers of deep learning architectures are often unimodal [1], making the mean a reasonable estimate for high-probability regions. Additionally, our simulation results confirm that blocking feature variations around the mean does not significantly affect accuracy. Therefore, we utilize the feature mean in the current version of MeanSparse. More efficient versions of MeanSparse in terms of robustness could be designed in future work by more precisely analyzing the distribution or considering multidimensional distributions instead of one-dimensional ones.

$**2-**$ For each model, we used the training set of the dataset on which the model was trained. We chose benign (non-adversarial) data from the training set because it provides a more reliable estimation of the mean and variance across different features compared to the validation or test sets. This ensures that the computed statistics accurately reflect the typical feature distributions.

$**3-**$ MeanSparse aims to suppress features near the unconditional mean of the distribution, without incorporating class labels into its computations. In other words, treating the transformation from input to features as an implicit generator, it empirically estimates the mean and variance of each feature independently of class labels.

[1] Shwartz-Ziv, R. and Tishby, N., 2017. Opening the black box of deep neural networks via information. arXiv preprint arXiv:1703.00810.

Thank you for your detailed feedback and valuable suggestions. Below, we address your concerns:

$**Gradient Masking Concerns:**$

Thank you for your thoughtful comment. We understand the concern regarding potential gradient masking and the need to test adaptive attacks, such as BPDA, to evaluate the robustness of MeanSparse-integrated models more thoroughly. Below is our detailed response:

$\bullet *Potential Gradient Masking:*$

The MeanSparse operator does induce zero gradients for the features it projects, which could limit the effectiveness of gradient-based attacks. To address this, we acknowledge that adaptive attacks, such as BPDA, where the projection operation is removed during gradient computation, could potentially reduce the robust accuracy. This limitation will be explicitly highlighted in the revised manuscript to ensure transparency.

$\bullet *Standardized Comparison with AutoAttack:*$

While adaptive attacks are valuable for understanding specific weaknesses, we used AutoAttack for all evaluations to maintain a standardized and comparable metric. AutoAttack is widely used for robustness evaluation and does not assume knowledge of the defense, ensuring consistency across models before and after integrating MeanSparse. Importantly, this also aligns with how models are evaluated in RobustBench.

$\bullet *Preliminary Adaptive Attack Analysis:*$

To explore the effect of adaptive attacks, we tested replacing the gradient of MeanSparse with an identity function (similar to BPDA). This approach, which modifies the backward pass, led to a decrease in AutoAttack accuracy. However, we recognize that further adaptive attacks (e.g., tailored BPDA) could reveal additional limitations, and we have included this discussion in Section 4.4 of the revised manuscript. By combining a standardized evaluation with initial adaptive attack experiments, we aim to provide a comprehensive analysis.

$**Discussion in Section 3.1:**$

In Section 3.1, we provided detailed explanations and intuition behind the proposed MeanSparse method. While Section 3.1 may initially appear disconnected from the final approach, its relevance becomes clear upon reading Section 3.2, where the connection to the exact MeanSparse formulation is established. $Z_{k-1}$ in Eq. (7) is mistakenly written instead of $\bar{a}_{k-1}$. Eq. (7) will be corrected in the revised version of the manuscript.

We appreciate your detailed review. Below, We address your concerns to enhance clarity and quality of our work:

$**Vague explanation:**$

Thank you for highlighting the inconsistency. We have revised the Introduction to clearly explain how MEANSPARSE integrates activation function design with feature sparsification to enhance adversarial robustness and provide a unified description of our contributions.

$**Lack of intuitive explanation:**$

Thank you for highlighting the need for an intuitive explanation of the "Mean-based Sparsification" method. Here, we have provided a concise explanation with a simple example and will include it in the Introduction of the revised version.

$*Explanation:*$

The MeanSparse operator selectively suppresses variations around the mean of feature representations, effectively filtering out non-robust features. For a given feature channel, we compute the mean ($\mu$) and standard deviation ($\alpha$) over the training set. Using a tunable threshold ($Th=α⋅σ$), we block feature values that lie within $μ±Th$, replacing them with the mean value ($μ$). This operation limits minor perturbations that adversarial attacks often exploit, while preserving the informative structure of features outside this range.

For instance, consider a hypothetical feature channel with a mean ($μ$) of 0.5 and standard deviation ($σ$) of 0.2. Setting $α=1$, we block values between 0.3 and 0.7, replacing them with 0.5. This simple mechanism attenuates insignificant variations, as demonstrated in Figure 1 of the paper, where we visualize how the input histogram is transformed. The blocked region corresponds to low-information variations, enhancing robustness by reducing the attacker's exploitable capacity.

$**Writing errors:**$

Thank you for pointing this out. We will revise the caption of Figure 2 to improve clarity.

$**Questions:**$

$**1-**$ The primary motivation behind MeanSparse is to block uninformative features from the model, thereby reducing the space available to potential attackers. Below is a clear comparison of the two approaches which have been included in Section 3.3 of the revised manuscript:

$\bullet **During Training:**$

$*Advantages:*$ MeanSparse influences the model's learned representations from the start, potentially improving robustness throughout the training process.

$*Disadvantages:*$ Requires a carefully designed threshold adjustment scheduler:

A rapid threshold increase disrupts training due to the gradient zeroing effect. A slow threshold increase mimics post-training behavior.

Aligning the threshold scheduler with activation functions is challenging, particularly for large models.

Difficult to scale to large models due to training instabilities.

$*Evidence:*$ Early experiments on smaller models often resulted in unstable training and failed convergence due to misaligned thresholds.

$\bullet **Post-Training:**$

$*Advantages:*$

Statistics of the model are already established, simplifying integration.

Only requires a search over alpha values, making it scalable to large models.

Successfully applied to models like Swin-L, leading to a +2.56% improvement in robustness with no destabilization.

$*Disadvantages:*$ Cannot influence learned representations during training.

$*Evidence:*$ Experimental results show that post-training integration consistently improves robustness without compromising performance, even in large-scale architectures.

Due to the challenges of training large models with MeanSparse, we opted for the more effective post-training integration approach.

$**2-**$ Thank you for your feedback. The experimental settings used for ablation study are:

Dataset: CIFAR-10

Architecture: ResNet-18

Optimizer: SGD (learning rate: 0.1, weight decay: 0.0005, momentum: 0.9)

Number of Epochs: 200

Batch Size: 256

Learning rate scheduler: The initial learning rate of 0.1 is reduced by a factor of 10 at epochs 100 and 150.

Best model selection: We evaluate the model at each epoch of training and select the one with the highest PGD adversarial accuracy on the test set.

Adversarial training properties: 10-step PGD adversarial training [1] with respect to $\ell_{\infty}$ attacks with a radius of 8/255 and step size of 0.0078.

The experiments were conducted on an NVIDIA A100 GPU, with model training taking about 6 hours and evaluation around 20 minutes per model.

The experimental details are provided in Appendix A.1. We will ensure to highlight this in the main body of the paper in the revised version for better visibility. Additionally, the code is publicly available at https://anonymous.4open.science/r/MeanSparse-84B0/ to facilitate reproducibility.

[1] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018.