Your comments are highly appreciated, and we have addressed them in this revision. $\color{red}{\text{All modifications are shown in red in this updated document.}}$

W1: I am not convinced that the model trained on some carefully selected and augmented data can be robust to adversarial attacks. Unlike adversarial training, this method does not rely on any specific adversarial attack. Therefore, it is suggested to conduct a more comprehensive evaluation of your method on different attacks, such as l2-AA [1], l1-AA [2], and sAA (l0-bounded perturbations) [2], compare the results to baseline methods to further demonstrate the effectiveness of your method.

Thank you for your question. Tables D1,D2 and D3 show the result of CIFAR-10 on ResNet-18 attacked by $l_2$-AA[1], $l_1$-AA[2] and s-AA[3]. Our algorithm can successfully improve the robustness of the model under these different adversarial attacks. More results are shown in $\color{red}{\text{Appendix F Table 6 and 7}}$.

Table D1: CIFAR-10 on ResNet-18 with $l_2$-AA.

Table D2: CIFAR-10 on ResNet-18 with $l_1$-AA.

Table D3: CIFAR-10 on ResNet-18 with s-AA.

[1] Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. ICML, 2020.

[2] Francesco Croce and Matthias Hein. Mind the Box: l1-APGD for Sparse Adversarial Attacks on Image Classifiers. ICML, 2021.

[3] Xuyang Zhong, Yixiao Huang, and Chen Liu. Towards Efficient Training and Evaluation of Robust Models against l0 Bounded Adversarial Perturbations. ICML, 2024.

W2: Line 58-59: If I understood correctly, FSE-Net is used to select important frequency components, so it is more like a data augmentation or purification method. However, it does not reduce the training samples or the size of the images. In this regard, why do you claim that FSE-Net can reduce storage requirements? You should provide a detailed explanation or quantitative analysis of how selecting frequency components translates to reduced storage needs.

Thank you for bringing up this question. Our method can indeed reduce the storage requirements. By applying Discrete Cosine Transform (DCT) and leveraging the sparse nature of the frequency domain, we store only significant non-zero coefficients using an optimized sparse storage format. Each non-zero coefficient requires 6 bytes: 4 bytes for the float32 value and 2 bytes for packed indices. Since the image size is 32×32, we can efficiently encode both row and column indices using 5 bits each, combining them into a single 16-bit integer. The actual compression results are shown in Table D4. The compression and depression processing is lossless and the IDCT reconstruction is efficient, taking only 3.43 seconds for CIFAR-10 with GPU. We have added this table to $\color{red}{\text{Appendix D Table 5.}}$

Table D4: Frequency pruning and storage reduction.

Author Response 2

3.1. Taylor expansion calculation: small perturbation

The Taylor expansion of a function around a point provides a polynomial approximation through an infinite sum of terms. In our analysis of model behavior under input perturbations, we utilize this expansion up to the second order, as higher-order terms become negligible for small perturbations $\delta_x$. Specifically:

where $\xi \in [0,1]$, $L(\theta, x, y)$ is the original loss value represents the base value before perturbation, $\nabla_x L(\theta, x, y)^\top \delta_x$ is the first-order approximation which is the inner product of gradient and perturbation, $\frac{1}{2}\delta_x^\top \nabla_x^2 L(\theta, x + \xi\delta_x, y)\delta_x$ is the second-order approximation which captures the local curvature of the loss landscape and $O(‖\delta_x‖^3)$ contains all terms of order 3 and higher. By keeping terms up to second order and neglecting higher-order terms due to small $\delta_x$, we have:

To formally characterize the boundedness of second-order derivatives, we adopt the notion of Lipschitz smoothness from [e]. Specifically, a function $f : \\mathbb{R}^n \\to \\mathbb{R}$ is defined to be $\\beta$-smooth if its gradient is Lipschitz continuous:

We used well-designed Neural Networks with finite layers and bounded parameters which implies that for $x_0 \in \text{supp}(\mathcal{X})$ and $x \in B(x_0,\epsilon)$, the Hessian matrix $‖\nabla^2_x L(\theta,x,y)‖_2 \leq \beta$ exists almost everywhere and satisfies:

Then we can have:

Therefore, taking expectation over the data distribution:

where $‖ \cdot ‖_p$ and $‖ \cdot ‖_q$ are dual norms satisfying $\frac{1}{p} + \frac{1}{q} = 1$. In this proof, we choose $p = q = 2$ which can make analysis of gradient and Hessian easier (from Cauchy-Schwarz inequality). This bound characterizes how the expected distortion depends on both the average gradient magnitude and the curvature of the loss landscape across the data distribution.

3.2. Integral Approximation: non-small perturbation

Using Taylor expansion is suitable when $\delta_x$ is small enough that we can ignore higher-order terms. However, when $\delta_x$ is not small enough, the effect of higher-order terms can't be ignored which will make Taylor's theorem not suitable so we need to use another formula that:

This formula can be used in our tasks because the loss function $L(θ, x, y)$ is continuous on the interval [$x, x + \delta_x$]. In order to avoid the path dependence of this formula, we utilize the Mean Value Inequality for vector-valued functions:

where $‖\cdot‖_p$ and $‖\cdot‖_q$ are dual norms satisfying $\frac{1}{p} + \frac{1}{q} = 1$, and conv($x, x + \delta_x$) represents the convex hull between points $x$ and $x + \delta_x$. This inequality allows us to bound the loss difference using the maximum gradient norm over the convex hull between $x$ and $x + \delta_x$, eliminating the need to consider specific paths.

Therefore, the Expected Distortion Rate can be bounded as:

where $B(x,\epsilon)$ denotes the ball centered at $x$ with radius $\epsilon$. This bound remains valid for infinitesimal perturbations $\delta_x$. However, the absence of second-order information (Hessian matrix) limits its ability to provide fine-grained characterization of the local geometry compared to the Taylor expansion bound, particularly in the regime of small perturbations where local curvature information becomes crucial for precise estimation.

Author Response 1

Thank you for the detailed elaboration and clarification of your concerns. I understand the importance of grounding our claims in rigorous theoretical analysis, the details of our proof are shown as follows:

Smooth Loss Landscape and Adversarial Robustness Analysis

To rigorously establish the relationship between a smooth (flat) loss landscape and higher adversarial robustness, we begin by defining the adversarial robustness measure in terms of the "Expected Distortion Rate (EDR)" of the loss function with respect to input perturbations. By connecting this metric to the gradient norm of the loss and demonstrating how smoother loss landscapes yield smaller gradient norms, we can show that a flatter landscape reduces such distortion. This ultimately supports the conclusion that smoother loss landscapes enhance adversarial robustness by limiting variations in loss under input perturbations.

1. Adversarial Robustness Measure: Expected Distortion Rate (EDR)

In this part, we start with the constant of the loss function with respect to input perturbations called "Expected Distortion Rate (EDR)", which quantifies the adversarial robustness. The definition is:

Let $\\mathcal{X}$ be a compact subset of $\\mathbb{R}^n$. Given a model parameterized by $\\theta$ with loss function $L(\\cdot)$, we evaluate its performance against ground truth label $y$. Here $|\\cdot|$ and $‖\\cdot‖_2$ denote the absolute value and Euclidean norm respectively, and we assume $x + \\delta_x \\in \\mathcal{X}$ for all $x \\in \\mathcal{X}$ to ensure perturbed inputs remain in the domain.

The perturbation from different kinds of adversarial attacks can be measured in $L_1$, $L_2$, or $L_\infty$ norms. We specifically use $L_2$ norm $‖\delta_x‖_2 \leq \epsilon \quad \text{for } \epsilon > 0$.

This definition encapsulates the core of adversarial robustness by quantifying the average sensitivity of the model's loss to adversarial perturbations across the data distribution. A smaller $\text{EDR}_\theta$ indicates that adversarial attacks have minimal impact on the model's performance.

2. Analysis of Loss Landscape Smoothness via First and Second-Order Geometric Properties

Based on the loss landscape visualization depicted in Figures 2 (a) (b) (c) and 3 (a), we observe that the geometric characteristics of the loss landscape are predominantly determined by two fundamental components. As we set the loss function $L(\theta, x, y)$ in the neighborhood of point $x_0$, the local behavior can be characterized through:

1. The gradient vector $\nabla_x L(\theta, x_0, y_0) \in \mathbb{R}^n$, representing the first-order derivative of the loss function, determines both the direction of steepest ascent and the local rate of change in the loss surface at $x_0$.
2. The Hessian matrix $\nabla_x^2 L(\theta, x_0, y_0) \in \mathbb{R}^{n\times n}$, which encapsulates the second-order derivatives, quantifies the local curvature and determines the rate at which the gradient changes in different directions around $x_0$.

Therefore, to achieve a smoothly varying loss landscape, it is necessary to simultaneously minimize both the magnitude of the gradient and the spectral norm of the Hessian matrix.

Prior research has proposed diverse metrics to characterize the smoothness of loss landscapes and their correlation with model generalization, including Volume $\varepsilon$-Flatness [a], Hessian-based measures [b] [c], and gradient-based analysis[d]. Our work adopts a more comprehensive approach by jointly analyzing both gradient and curvature characteristics across extended regions of the loss surface. This broader perspective is particularly vital for understanding adversarial robustness, as adversarial perturbations can push model predictions far from local minima, where the geometric properties of non-minimal regions become crucial determinants of model behavior.

3. Relating the EDR to the Gradient and Hessian matrix

The relationship between adversarial robustness and loss landscape smoothness can be analyzed through two complementary situations, depending on the magnitude of perturbation $\delta_x$:

1. When $\delta_x$ is sufficiently small such that higher-order terms can be neglected, we employ second-order Taylor expansion to obtain tight bounds (Section 3.1).

2. For larger $\delta_x$ where higher-order terms become significant, we utilize integral approximation to derive more appropriate bounds (Section 3.2).

This dual approach allows us to comprehensively characterize the relationship between loss landscape smoothness and adversarial robustness across different perturbation regimes.

In short, it is true that minimizing adversarial loss can achieve non-trivial robustness and smoothing adversarial loss function can improve robustness. Smoothing arbitrary loss function is unlikely to obtain non-trivial robustness. My question is why minimizing the loss on frequency-pruned images can also achieve non-trivial robustness？

Thank you for taking the time to review our work! We have incorporated your valuable input, and $\color{red}{\text{the changes can be easily identified in red in this updated version.}}$

W1: Line 187-188: "Frequency pruning removes textural details while preserving key shape features, helping the model focus more on shape, as shown in Fig. 1." Figure 1 does not clearly show this; how were these examples selected? Could you better explain how this figure demonstrates how it makes the model focus more on shape than textural details?

Thank you for your valuable question.

1. No clear Pattern? The green highlights in Figure 1 show network attention patterns. Compared to the last column (original dataset), the middle column (ours) better preserves the entire shape of objects.

2. How images selected? The images were randomly selected from the ImageNet-1K training set, and we display only three due to space limitations.

3. Why shapes over textures? Frequency pruning preserves global shape features while removing textural details and noise, encouraging the model to prioritize structural information over local textures. This is the motivation for using frequency pruning to enhance model robustness.

W2: Figure 2 is qualitative, mainly as smoothness is not defined; I presume that the smoothness of the overall landscape is from -1 to 1 (x-axis). This could be significantly improved by providing a quantitative measure for smoothness to allow for more comparable measures. For example the average gradient magnitude or curvature.

Thank you for your constructive feedback and construction.

1. Qualitative Analysis. Visually, smoothness manifests as gentle, rounded curves overall landscape is from -1 to 1 (x-axis).

2. Quantitative Analysis. We calculate the Average Gradient Magnitude (AGM) and Average Curvature (AC) of three lines and show these results in Table C1. Lower AGM and higher AC refer to better smoothness.

Table C1: Average Gradient Magnitude and Average Curvature

W3: These loss landscape figures are not averaged, and given that, there can be quite different results depending on the random directions used to generate the plots. See A.4 in [1]. Plotting the average loss landscape from 5 runs along with the standard deviation would help further the position that smoothness and adversarial robustness are linked.

Thank you for your suggestion. We have updated the plots of 5 runs in $\color{red}{\text{Appendix H Figure 9}}$. While the exact loss landscapes vary across 5 independent runs, the relative smoothness characteristics between different methods remain consistent, validating the reliability of our comparative analysis.

W4: Throughout the paper, averages and standard deviations are missing; this significantly hampers the results and does not follow best practices. Could you provide an average from 5 runs?

Thank you for your valid points. Table C2 provides the average of our experiment from 5 independent runs with the std indicated. More experiment results are added to $\color{red}{\text{Appendix G Table 9}}$.

Table C2: CIFAR-10 performance under various adversarial attacks and dataset pruning ratio. Results are computed over 5 independent runs.

W5: Do you have a similar table to Table 2 for CIFAR-100?

Thank you for the suggestion. We have provided the comparison for CIFAR-100 in $\color{red}{ \text{Appendix K Table 14 (c).}}$ Table A2 shows that our algorithm performs better than other adversarial training (AT) algorithms when we prune the coreset with 50% pruning ratio in terms of both time and accuracy.

Table A2: Comparison of our algorithm with different adversarial training (AT) algorithms

W6: Have you considered an adaptive attack?

Thank you for the question. Table A3 shows our algorithm has high robustness to adaptive attack $\ell_1$-APGD attack [b], more results are updated in $\color{red}{\text{Appendix F Table 8}}$.

Table A3: $\ell_1$-APGD attack with different pruning ratios on CIFAR-10 ResNet18.

[b] Francesco Croce and Matthias Hein. Mind the Box: l1-APGD for Sparse Adversarial Attacks on Image Classifiers. ICML, 2021.

W7: For larger datasets would the LF pruning approach still be feasible?

Thank you for your question. In this experiment, we tested the ImageNet-1K dataset, which contains over 1 million 224-resolution images. ImageNet is one of the largest datasets commonly used in dataset pruning studies, as shown in [c][d][e].

[c] Haizhong Zheng et al. Coverage-centric Coreset Selection for High Pruning Rates. ICLR,2023.

[d] MariyaToneva et al. An Empirical Study of Example Forgetting during Deep Neural Network Learning. ICLR, 2019.

[e] Geoff Pleiss et al. Identifying Mislabeled Data using the Area Under the Margin Ranking. NeurIPS, 2020.

We deeply appreciate your constructive comments, as they have helped us improve the overall rigor and quality of our manuscript.

Thank you for your insightful comments and kind words. We have reviewed them carefully and made the necessary adjustments. $\color{red}{\text{Key changes and additions are marked in red throughout this revised document.}}$

W1: Are any of the following considered as closely related work?

• Adversarial Coreset Selection for Efficient Robust Training (IJCV/ECCV'22) https://arxiv.org/abs/2209.05785
• Less is More: Data Pruning for Faster Adversarial Training (AAAI-23 workshop, SafeAI) https://arxiv.org/abs/2302.12366
• Adversarial training with informed data selection (EUSIPCO'22) https://arxiv.org/abs/2301.04472
• Efficient Adversarial Contrastive Learning via Robustness-Aware Coreset Selection (NeurIPS'23) https://arxiv.org/abs/2302.03857

Thank you for your question. These works focus on reducing the costs of Adversarial Training (AT), and their algorithms have goals and application scenarios that differ from ours. The key differences are:

1. Their algorithms still incur training costs during AT, whereas ours requires no additional training cost, making theirs unsuitable for resource-limited environments because their algorithms also need extra training costs.

2. These algorithms address the adversarial robustness of the entire dataset, while our work focuses on robustness within a pruned dataset.

W2: There is also the Sabre work: https://alec-diallo.github.io/publications/sabre Is this related as it performs a spectral decomposition of the input and selective energy-based filtering? (https://arxiv.org/html/2405.03672v3)

Thank you for your attention to the comparison between our algorithm and other algorithms. Our algorithm and Sabre [a] have the following essential differences:

1. We employ a learnable frequency pruning method to minimize model entropy, rather than selecting frequency components solely based on energy ranking which highlights the principles of adversarial robustness. On the other hand, Sabre [a] focuses on recognising and removing non-robust features which makes our core idea different.

2. Sabre [a] does not lead to a decrease in the dataset size, while we conduct experiments to validate the effectiveness of dataset pruning.

[a] Alec F. Diallo and Paul Patras. Sabre: Cutting through Adversarial Noise with Adaptive Spectral Filtering and Input Reconstruction. IEEE Symposium on Security and Privacy, 2024.

W3: Could you comment briefly on the overall practicality of the technique, accuracy is still very low vs. adversarial samples?

Thank you for your valuable feedback.

1. Training models on a pruned dataset in resource-limited edge devices is a highly challenging task. For instance, even with NO attack, ResNet-18 drops to 86% (compared to previously 95%) after 90% pruning ratio on CIFAR-10. Achieving robustness is even harder, and due to the high computational cost of adversarial training, our method provides a feasible solution. In addition, Table 1 has shown our method surpasses other attacks by a large margin.

2. This work is the first work studies the robustness of pruned datasets, laying the foundation and providing a strong baseline for future research.

W4: When using LF (Ours-LF) to preserve 50% of the frequency components, what are the actual reductions in training data storage requirements results?

Thank you for bringing up this question. By applying Discrete Cosine Transform (DCT) and leveraging sparse nature of the frequency domain, we store only significant non-zero coefficients using an optimized sparse storage format. Each non-zero coefficient requires 6 bytes: 4 bytes for the float32 value and 2 bytes for packed indices. Since the image size is 32×32, we can efficiently encode both row and column indices using 5 bits each, combining them into a single 16-bit integer. The actual compression results are shown in Table A1. The compression and depression processing is lossless and the IDCT reconstruction is efficient, taking only 3.43 seconds for CIFAR-10 with GPU. We have added this table to $\color{red}{\text{Appendix D Table 5.}}$

Table A1: Frequency pruning and storage reduction.

We greatly appreciate your detailed feedback! Your suggestions have been taken into account, and $\color{red}{\text{the updates are highlighted in red to ensure clarity in this revision.}}$

W1: There is no explicit definition of Joint Entropy. Does it refer to a combination of multiple scores (such as EL2N, GraND, etc.)?

Thank you for the question. It is not a combination of multiple existing scores. Simply put, "Joint Entropy" is measuring how uncertainty changes and connects across time. Therefore, our method brings different training periods together to understand their combined patterns of change.

W2: The definition of Reward Score on page 6 is confusing. What is its relationship to Joint Entropy? What is the exploration phase? What is the exploitation phase?

Thank you for your valuable feedback. We would like to clarify, that the Reward score evaluates the variation of each image throughout the training process. It is calculated as a cumulative score over the entire training period using Eq. (10) and (11). The two phases are:

• Exploration Phase: During the first ⅔ of training, the model explores the feature space, developing diverse feature representations and broad learning capabilities.

• Exploitation Phase: In the final ⅓ of training, the model focuses on exploiting learned features to optimize within a stable loss landscape, rather than exploring new feature spaces.

W3: Does the effectiveness vary with different optimization goals and loss curvature optimization (e.g., "Adversarial Weight Perturbation Helps Robust Generalization, NeurIPS 2020", "Theoretically Principled Trade-off between Robustness and Accuracy, ICML 2019")?

Thank you for the suggested comparison. Table B1 has been updated in $\color{red}{\text{Appendix K Table 14(a)}}$, and our algorithm delivers better performance than AWP [a] and TRADES [b].

Table B1: Comparision with AWP and TRADES on CIFAR-10 with PGD-20 attack

[a] Dongxian Wu et al. Adversarial Weight Perturbation Helps Robust Generalization. NeurIPS, 2020.

[b] Hongyang Zhang et al. Theoretically Principled Trade-off between Robustness and Accuracy. ICML, 2019.

Thank you for your questions and suggestions! We would like to address your concerns one by one. $\color{red}{\text{Additional materials and updates are highlighted in red in this PDF revision.}}$

W1: Lemma 1 is critical, but in the derivation, the assumption that $\frac{\partial f_\theta(\tilde{x})}{\partial \theta}$ is bounded is overly restrictive and lacks supporting evidence, requiring a more rigorous mathematical proof.

Thank you for your question. The boundedness assumption is naturally guaranteed by modern deep learning architectures, as it is an inherent property of network design rather than an additional constraint. Specifically:

1. Network architectures inherently prevent gradient explosion through weight initialization schemes and bounded activation functions [a].
2. Modern architectural components like residual connections [b] and batch normalization [c] are specifically designed to maintain gradient stability throughout training. Therefore, the boundedness assumption in Lemma 1 directly follows from standard architectural design principles rather than being an additional restriction.

[a] Razvan Pascanu et al. Understanding the exploding gradient problem. https://arxiv.org/abs/1211.5063v1.

[b] He et al. Deep residual learning for image recognition. CVPR, 2016.

[c] Sergey Ioffe et al. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. https://arxiv.org/abs/1502.03167.

W2: Appendix E lacks detailed structural information, providing only the structure names.

Thank you for your comment. The detailed structural information is:

Model Structure:
SEBlock(
    (avg_pool): AdaptiveAvgPool2d(output_size=1)
    (fc): Sequential(
        (0): Linear(in_features=N, out_features=N/16, bias=False)
        (1): ReLU(inplace=True)
        (2): Linear(in_features=N/16, out_features=N, bias=False)
        (3): Sigmoid()
    )
)

We have updated the detailed structural information to $\color{red}{\text{Appendix E}}$.

W3: Pseudo-Code in the second line has a missing definition for (\hat{X}_{i,j}).

Thank you for your valuable feedback. We have added the definition for $\hat{X}_{i,j}$ in Algorithm 1, where as mentioned in line 230,

$\hat{X}_{i,j}$ represents the $(i,j)$-th coefficient of the Discrete Cosine Transform (DCT) of an image $X$.

W4: Equation 6 applies derivatives only to correctly classified samples. If the count of correct classifications decreases during updates, can incorrect samples still guide parameter updates? Otherwise, the function will only reinforce correct predictions without helping incorrect ones become correct.

Thank you for your insightful feedback.

1. Equation 6 minimizes the model's entropy while maximizing classification accuracy by updating all samples, with correctly classified samples receiving an additional reward.
2. We set $\lambda=0.1$ (see Figure 4(b)) before applying the correctly classified reward to ensure that $\lambda \left( \frac{1}{|\mathcal{D}|} \sum_{(\tilde{x},y)\in\mathcal{D}} \mathbf{1}{\arg \max f_{\theta}(x_f)=y} \right)$ does not dominate the total loss, allowing error samples to receive effective gradient updates primarily through the first term.

W5: The terms "high energy" and "low energy" appear for the first time without a precise definition.

Thank you for your attention to the definition.

1. The definition of "high energy" is provided in lines 251–252: $E^{(k)}$ represents the energy of the $k$-th highest frequency component and high energy refers to components with energy greater than $E^{(k)}$.

2. In addition, an ablation study on the choice of $k$ is shown in Figure 4(c).

W6: Section 3.3 lacks clarity on how it integrates with previous algorithms (requires a flowchart or equivalent explanation)

We deeply appreciate your insights on our methodology.

The flow chart for our algorithm is as follows:

A[Input Images] --> B[Frequency Domain Transform]
B --> C[Learnable Component Selection]
C --> D[Inverse Transform]
D --> E[JE Score Calculation]
E --> F[Score-based Ranking]
F --> G[CCS Sampling]
G --> H[Final Coreset]

The illustration diagram of the flow has been added to $\color{red}{\text{Appendix B Figure 6.}}$

We sincerely thank you for your valuable feedback, which has significantly enhanced the clarity and quality of our paper.

Dear Reviewer HAQ8,

We greatly appreciate the time and effort you've invested in reviewing our work. As we are now less than one day away from the discussion deadline, we wanted to reach out regarding our rebuttal, which we submitted earlier.

If possible, we would be grateful for any feedback you might be able to provide, as this would give us the opportunity to engage in a productive discussion before the deadline. We're eager to address any remaining questions or concerns you may have about our paper.

We understand that you likely have a busy schedule, and we truly appreciate your valuable insights and expertise in this process.

Thank you for your time and consideration.

Dear Reviewer HAQ8,

I apologize for reaching out again after my previous message. With the discussion deadline now just one day away, I wanted to very gently follow up on our earlier correspondence regarding our paper and rebuttal.

We completely understand if you haven't had the opportunity to review our rebuttal yet, given the many demands on your time. However, if you have had the chance to look it over, we would be immensely grateful for any feedback you could provide. Your insights, no matter how concise, would be valuable in helping us engage in a constructive discussion before the deadline closes.

We are sincerely thankful for your continued involvement in this process, and we respect your time and professional commitments. Thank you for your understanding and consideration.