DataFreeShield: Defending Adversarial Attacks without Training Data

We thank the reviewer for taking the time to review our work. We sincerely address their comments below.

It is not clear how you synthesize the samples. Especially, how can you adopt Eq (8) to generate the samples?

$\to$ In essence, random Gaussian noise is fed to a pretrained teacher, and its output/intermediate values are used to compute the loss terms in Eq (8). Then, the computed loss is backpropagated with respect to the input noise, where the pixels are updated using Adam optimizer. Please also see the red arrows depicted in figure2, and the pseudocode in section B of the appendix for more details.

It is not clear why the authors can adopt synthetic data to improve the model robustness while the images from other domains cannot. In my opinion, the synthetic data is also from different domains.

$\to$ Because synthetic data is always available, while images from other domains are not. To clarify, it is not the matter of whether the data is in-domain or not, but whether the dataset is available or achievable when only a pretrained weight is given. The term “data-free” used in our problem setting, similar to data-free quantization [1-3] and data-free knowledge distillation problems [4,5], means one is only given a pretrained model with no access to its training data. Thus, under this “data-free” constraint, synthetic dataset is always accessible (because we generate them) while other domain dataset is not guaranteed.

[1] Nagel, M. et al. Data-free quantization through weight equalization and bias correction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019.
[2] Cai, Y. et al. ZeroQ: A novel zero shot quantization framework. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
[3] Xu, S. et al. Generative low- bitwidth data free quantization. In European Conference on Computer Vision, 2020.
[4] Nayak, G.K. et al. "Zero-Shot Knowledge Distillation in Deep Networks. In International Conference on Machine Learning, 2019.
[5] Chen, H. et al. DAFL:Data-Free Learning of Student Networks. In Proceedings of the IEEE International Conference on Computer Vision, 2019.

There might be other ways that do not need training data to defend against adversarial attacks.

$\to$ It is important to note that while different adversarial defense techniques have been suggested in the past, only adversarial training(AT) and its variants have survived. Others have been shown to be ineffective or weak, often unable to defend against strong attacks. For example, multiple quantization and randomization techniques have been proposed as viable defense methods, but were quickly deprived of their efficacy and found to be a false sense of defense that can easily be circumvented [1]. For now, AT has become the dominant approach that is shown to be most effective. Thus we adopt the same AT approach in our problem. To support our argument, we provide comparison against test time defense methods, DAD and TTE, in the table below. Notice how they are not competitive against ours in all cases. We will add the following results to Table2 in the revised manuscript.

[1] Athalye, Anish, Nicholas Carlini, and David Wagner. "Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples." International conference on machine learning, 2018.

How to extract robust knowledge from clean images is not clearly expressed in this paper. Why can adversarial robustness be obtained using only loss constraints?

$\to$ In our method we do not extract robust knowledge from clean images. Instead, we extract knowledge from a pretrained teacher to generate synthetic samples, which are then used in adversarial training. To be specific, we use a pretrained network as guidance to optimize pixels from random Gaussian noise to image-like distributions. First, a set of random Gaussian noise is forwarded through the teacher network. Then, a set of optimization terms is computed using its output and/or intermediate features and statistics. Lastly, the computed loss is backpropagated all the way up to the input, where the gradient is used to update each pixel using Adam optimizer. Then, the generated samples are used in adversarial training.

If this is the case, can it be used in any confrontation training?

$\to$ Yes. As shown in Table 6 of the main manuscript, the set of generated images can be used in any adversarial training methods that require a set of input images. (Is ‘confrontation’ referring to adversarial? If not, please clarify the meaning of “confrontation”.) The table shows results for using different variations of adversarial training methods with our generated images.

What is the relationship between adversarial robustness and the amount of generated data?

$\to$ As described in Appendix C, adding more generated training samples increases the robust accuracy, but saturates at some point (5-60,000 samples for CIFAR-10), where further increment in data size does not lead to meaningful performance boost. We have the sample quantity-accuracy plot in Appendix C for the suggested study. Thank you!

We thank reviewer RMmq for the encouraging remarks on novelty and the presentation of our work. We address your concerns as follows.

Compared with the adversarial training model, the adversarial robustness obtained in Table 3 is very low and difficult to use in practice.

$\to$ We respectfully disagree with the reviewer’s comment on the amount of robustness and the practicality of the proposal. The reviewer compares our results against the conventional approach where the original train set is available. We gently remind the reviewer that the purpose of studying this particular setup is not to propose data-free approach’s superiority against data-driven ones, but to provide an alternative solution when none is available. (And currently, there is no known solution to make a neural network robust without using the original dataset.) We provide a few examples from other tasks to show that data-free methods have an inevitable gap with data-driven ones. For knowledge distillation, data-free approach[1] compares up to -10% to the data-driven one by Hinton[6]. This gap is larger for quantization, where data-free approaches show degradation up to -42.67% compared to data-driven quantization work[7] of the same year.

Table 1 : Data-free vs Data-driven in Knowledge Distillation

Table 2 : Data-free vs Data-driven in 4-bit Quantization

[1] Nayak, G.K. et al. "Zero-Shot Knowledge Distillation in Deep Networks. In International Conference on Machine Learning, 2019.
[2] Chen, H. et al. DAFL:Data-Free Learning of Student Networks. In Proceedings of the IEEE International Conference on Computer Vision, 2019.
[3] Nagel, M. et al. Data-free quantization through weight equalization and bias correction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019.
[4] Cai, Y. et al. ZeroQ: A novel zero shot quantization framework. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
[5] Xu, S. et al. Generative low- bitwidth data free quantization. In European Conference on Computer Vision, 2020.
[6] Hinton, G. et al. Distilling the Knowledge in a Neural Network. In Advances in Neural Information Processing Systems, Deep Learning Workshop, 2015.
[7] Nagel, M. et al. Up or Down? Adaptive Rounding for Post-Training Quantization. In International Conference on Machine Learning, 2020.

Please analyze the time complexity of the compared methods.

$\to$ Time complexity for our method is O(N*(F+B)) for data synthesis and another O(N*(F+B)) for student training. The baseline methods take O(N*(Fg+Bg+F+B)) and O(Fg+F+B) for data synthesis and student training, respectively. We have omitted the number of iterations for each process for clarity. Let F,B denote time complexity for each forward and backward computation. In our method, data synthesis requires additional forward and backward computation for optimizing each batch of pixels. Using the same notations, the baseline methods take N*(Fg+Bg+F+B) where Fg and Bg denote forward and backward computation to train the generator. Each generator update requires forward and backward computation of the teacher model, hence (Fg+Bg+F+B). Then, (Fg+F+B) is required to generate the samples, and train the student model. We also report wall clock time in Appendix A.2, where our method takes 0.6 hrs on ResNet-20, 2.6 hrs on ResNet-56, and 3.6 hrs on WRN28-10. The compared methods (DaST, DFME, AIT, DFARD) take 1.5hrs using ResNet-20, 2.75hrs using ResNet-56, and 10hrs using WRN28-10 on RTX3090 under the same environment. The time is measured for generating 10000 synthetic samples on a single GPU.

In Table 2, for Derma => DFME, why are the A_clean and A_pgd numbers the same?

$\to$ Because most datasets in MedMNIST-v2 have uneven class distributions, and training could easily diverge to one of the classes. In the case of Derma, the class imbalance is severe, where each class contains [66, 103, 220, 23, 223, 1341, 29] samples, 2005 in total. The identical numbers reported in the table is due to the model failing to learn and simply diverging in random guessing one of these classes. For example, if the model diverges towards 5th class, the resulting accuracy is 11.12%(=223/2005), which is why ResNet-18 trained using DFME shows this number. And since the model has diverged, all the gradient-based attacks become ineffective and thus the numbers are the same across all settings (clean, PGD, AutoAttack). Thus the results only mean that DFME makes training unstable and difficult to converge when applied to our problem. To make a fair observation, we will revise the table to report F1 scores instead of naive accuracy. We hope this alleviates the reviewer’s concern towards the credibility of the experiment results.

Can you please explain the issues in Table 3 (listed in Weaknesses section)?

a. SVHN/DFARD/ResNet-20: A_pgd (weaker attack) is lower than A_aa (stronger attack).
b. SVHN/DaST/ResNet-56: A_pgd is lower than A_aa.
c. Cifar10/DFARD/WRN28-10: A_clean is lower than A_pgd.

$\to$ a & c: are possible outcomes of synthetic data training, and possible signs of gradient obfuscation[1]. The signs of gradient obfuscation include increasing attack strength not necessarily leading to better attack success rate, which is what we can observe from Table 3. We agree the phenomenon is rarely observed in real-data adversarial training, where the train dataset is reliable and does not contain any outliers in terms of distribution or quality. However, all the baseline methods are data-free (See Appendix A.4 for explanation on baseline methods and how we adapt them to our problem), and use synthetic data for training. Learning from these data is tricky and adding adversarial perturbation only makes it harder. Thus, the model often diverges, or simply learns to circumvent gradient-based attacks in an unintended way.

$\to$ b: We found that for SVHN ResNet56 DaST, there has been an error, where 20.20 was written as 0.20. We apologize for the mistake and causing confusion to the reviewers.

[1] Athalye, Anish, Nicholas Carlini, and David Wagner. "Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples." International conference on machine learning, 2018.

How does the proposed defense fare against adaptive attacks?

$\to$ We are to update the results using adaptive attacks.

It is not clear how the gradient refinement strategy will lead to smoother loss landscape. Also, how does the gradient refinement strategy and the third loss term differ in what they are trying to achieve?

$\to$ The two methods essentially hold the same objective, but achieves it in a distinct way. Their ultimate goal is to learn adversarial robustness from synthetic samples that can be applied to real attacks. The third term in the training loss penalizes the model when it is highly sensitive to small perturbations. Implicitly, this would make the model favor a set of parameters that have a smoother loss surface with respect to the input. On the other hand, the gradient refinement method explicitly ignores gradients of parameters that are highly fluctuating, pushing the model towards a smoother loss landscape where loss does not fluctuate with small changes in the parameters. The ablation study in Table7 shows that improvement from both the loss term and the gradient refinement method are distinct and best when used together.

We thank the reviewer for a thorough and detailed review of our work, and the encouraging remarks on the effectiveness of our methods, along with its presentation. We address the concerns and questions provided by the reviewer.

DAD does not require access to the original dataset, making it a data-free defense. How does the proposed defense compare against DAD and TTE?

$\to$ We respectfully argue that both DAD and TTE are not ‘data-free’ for the following reasons. DAD heavily relies on the existence of similar dataset (which we think is already not data-free), and it also uses the original test data to tune their model, which is a far extreme assumption and could be regarded as data leak. Also, it is difficult to view TTE as “data-free” because they propose TTE as an enhancement method applied to already robust models trained using the original data.

Despite the unfairness, we compare DAD and TTE with ours. DAD does not perform well and there exists a large performance gap to ours in all cases. Note that DAD is evaluated on the same test set that is used to finetune/adapt the detector module, and yet shows poor results. While TTE is advantageous in preserving clean accuracy, the overall robust accuracy is low and not comparable to ours. Both approaches do not directly train the target model, which makes it vulnerable to gradient-based attacks. Thus, comparison against test time defense methods only enhances our assertion that existing methods are insufficient to guarantee robustness when the train data is unavailable.

We used the official code provided by the authors. For DAD, we follow their method and retrain a Cifar-10 pretrained detector on MedMNIST-v2 test set. For TTE, where we used +flip+4crops+4 flipped-crops as it is reported as the best setting in the original paper. We will add this comparison to Table 2 in the revision.

In case of pre-training with Derma, why is adversarial training with organC better than doing so with Derma itself (figure 1, right)?

$\to$ Because OrganC dataset contains more train samples than Derma. For MedMNIST-v2, each dataset in the collection contains a different number of training samples. Derma provides 7,007 training samples while OrganC provides 13,940, which is nearly double. This is possibly the reason OrganC provides a slightly better performance than using Derma itself, due to having more adversarial samples to learn robustness from. Also, due to the nature of biomedical dataset, the environment in which the dataset was collected could differ between each dataset, leading to difference in quality of the collected samples. The “rare cases'' we say in our paper refer to these external causes that we usually do not have control over.

In figure 3, for fixed (b) and dynamic (c) coefficient methods, how often do blue points appear in red space and vice versa?

$\to$ (b) shows 2.8% and 6.1% misclassified points for each class, while (c) has 5.6% and 9.1%. The table below shows the exact numbers. For each plot we generated 2048 data points, where half is given label 0 (red) and the other half 1 (blue) when generating with cross entropy loss. So ideally, generated data points should have equal distribution in both regions of color.

In the results, we can see that (c) does show higher error, with 2.8% and 3.0% increase in each case. However, the difference is not large enough where it can hinder the model’s ability to learn discriminative features. On the other hand, the gained diversity is highly noticeable. The trade-off with discriminative features is not a counter productive factor but something we can leverage, where a small sacrifice in discriminative features leads to huge improvement in diversity.