Adversarial Defense using Targeted Manifold Manipulation

We appreciate our reviewer's time and effort to provide us with detailed feedback and raise a few interesting points. Please see below our replies to the key points raised.

• Stability of our defense

  Ans: To stress-test our defense we designed an adaptive attack that assumes the knowledge of the defense algorithm and the associated parameters to defeat the detection system. This should represent the worst of all the unknown attack strategies (sec 4.4). However, we showed that our detection methods still provided respectable accuracies against such extreme attack scenario.

• Q. Detection speed

  Ans: We have already covered that under the 'Computational time' in the experiment section (the last paragraph).

• Q. Loss function components

  Ans: Loss components are indirectly given weight by sampling the data instances corresponding to individual losses at different ratios (sec 4.1). Thus we don't need to provide individual weights in the combined loss function.

• Q. 3.4 Threshold values of $\rho$ and $\xi$

  Ans: We did specify a method to set the thresholds in sec 4.2, para 'Offline attack detection performance'. They are set to cause at most 0.5% false positive rate in the training dataset.

• Q. Uncertainty due to randomness of the trigger

  Ans: The randomness of the trigger (both the norm and the position) as discussed in sec 3.1 is by choice. Fig 3 shows the effect of our trigger placement. The trapring design as seen in there is only possible by randomly placing the trigger and the thick skin of the trapring is possible by using the random trigger norm. Thus they are the necessity for our method and not the negative aspects.

We appreciate our reviewer's time and effort to provide us with detailed feedback and raise a few interesting points. Please see below our replies to the key points raised.

• On Tiny imagenet sample count

  Ans: Sorry, it was a typo and your numbers are correct. Thank you.

• Use of higher learning rates during attack

  Table 1:Detection accuracies against attacks with different learning rates.

  -------------------------------------------------------------
  |   Lr   | Method| FGSM | PGD |  CW |   DF  |  PA   |   AA  |
  |------------------------------------------------------------
  | 1/255  | TMM-O |  100 |99.99|90.67| 99.98 | 98.50 |  100  |
  | 1/255  | TMM-L |  100 | 100 | 100 | 99.99 | 85.24 |  100  |
  | 2/255  | TMM-O |  100 | 100 |88.71| 99.98 | 97.31 |  100  |
  | 2/255  | TMM-L |  100 | 100 | 100 | 98.99 | 83.09 |  100  |
  | 4/255  | TMM-O |  100 | 100 |96.44| 98.34 | 98.49 |  100  |
  | 4/255  | TMM-L |  100 | 100 | 100 | 99.73 | 85.17 |  100  |
  | 8/255  | TMM-O |  100 | 100 | 100 | 99.97 | 98.57 |  100  |
  | 8/255  | TMM-L |  100 | 100 | 100 | 99.91 | 87.01 |  100  |
  | 16/255 | TMM-O |  100 | 100 |98.78| 99.13 | 97.09 |  100  |
  | 16/255 | TMM-L |  100 | 100 | 100 |  100  | 92.14 |  100  |
  -------------------------------------------------------------
  

  Ans: As we discussed in sec 3.1, the norm of our trigger is sampled between [$\epsilon$, 0.99], which means that the Trapring can be quite wide, sort of creating an effect of filling up the vacant spaces between the data instances with Traps. Thus, it is actually easier for the attacks to fall into the Trapring with higher learning rate (step-size) as shown in the below table.

• On the threat model and defender's knowledge about attack parameters

  Ans:We are extremely sorry for the confusion created due to overloading the same symbol ($\epsilon$) for both the trigger's lowest norm ($\epsilon_{training}$, used during training) and attack's perturbation budget ($\epsilon_{attack}$, used during attack). Trigger's lowest norm is set as small as possible so that clean accuracy is not hampered much. In our training, we used 0.01 (2.55/255) as the minimum $\epsilon$ for training. This means that any attack using a larger perturbation budget than 0.01 will end up in the Trapring. Attacks with smaller perturbation budget either fails because of extreme closeness requirement to the genuine data or still gets detected because even though the Traprings are enforced starting at 0.01, it starts to induce Trapring formation even before that. In short, the defender does not need to know the attacker's perturbation budget ($\epsilon_{attack}$) and it needs to use just the minimum perturbation ($\epsilon_{training}$) that it can use to avoid drastic reduction in the clean data performance, which can usually be very small. Thus, our method can provide defense across a wide spectrum of attack parameters.

• Q. CIFAR10.1 dataset results

  Ans:As suggested, we have conducted the experiment on CIFAR10.1 dataset. Given table shows the performance. We can see only 1.81% increment in the False Positive (FP) rate without any drop in the detection accuracy.

  ----------------------------------------------------------------------
  |   Dataset  | Method| FP | FGSM | PGD |  CW |   DF  |  PA   |   AA  |
  |---------------------------------------------------------------------
  | CIFAR10    | TMM-O |5.33|  100 |98.87|94.21| 99.98 | 98.24 |  100  |
  | CIFAR10    | TMM-L | 0.0|  100 | 100 |99.98| 99.80 | 81.54 |  100  |
  | CIFAR10.1  | TMM-O |7.14|  100 |98.69|96.24| 98.73 | 98.20 |  100  |
  | CIFAR10.1  | TMM-L | 0.0|  100 | 100 | 100 | 98.99 | 83.24 |  100  |
  ----------------------------------------------------------------------
  

• TMM on the point of view of Adversarial Training

  Ans:In the adversarial training (AT), we use adversarial samples to make the model robust. The adversarial training has two significant drawbacks: 1) the robustness obtained by AT depends on the attack methods used in the adversarial sample generation process. It has been shown that adversarial robustness against one attack method does not transfer to other attack methods. 2) AT can have large impact on the clean data accuracy as it can blur the class boundaries between genuine classes. In contrast, our method do not need to know the attack method as our design ensures universal defense and it has a least effect as instead of blurring the class boundaries it creates Traprings around the genuine data which has a very low impact on the clean data accuracy. Thus our method offers a superior choice.

We thank our reviewers' efforts to raise important points and provide valuable suggestions.

• The effect of data quality

  Table 1: Detection performance for model trained with datasets having varying quality.

  -----------------------------------------------------------------------------------------------
  |          Method   | Clean Acc.|  eps  |   FP  | FGSM |   PGD  |   CW   |   DF  |  PA   | AA |
  |-----------------------------------------------------------------------------------------------
  |    Clean Dataset  |   94.01   | 4/255 |  5.33 |  100 |  98.87  | 94.26 | 98.24 | 99.98 | 100 |
  |    Noisy label    |   89.41   | 4/255 |  3.40 |  100 |  98.43  | 80.21 | 89.98 | 85.57 | 100 |
  | Poison Adversary  |   90.61   | 4/255 |  4.59 |  100 |  97.57  | 78.39 | 85.36 | 82.36 | 100 |
  ------------------------------------------------------------------------------------------------ 
  

  Ans:We investigated the data quality issue against two different kinds of noise: label noise and poison data as in BadNet [1] on the CIFAR-10 dataset. For label noise we randomly selected 5% of the training data and then changed their labels to randomly selected other labels. For poisoning data we again randomly selected 5% of the training data, added a trigger of size 5x5 (solid blue) and made their class label as 5 ('Dog'). As we can see form the below table that even with such a high level noise the detection rates across different attacks remain useful. Poisoning data has more adverse effect than noisy labels. Even though we got high backdoor accuracy (~100%) with this poison dataset, for most of the data instances our Trapring still provided a sharper gradient, thus forcing adversarial attacks to go through the Trapring, causing detection. CW method suffered the most, but looking at the false positive (FP) rate it looks like we can tune the thresholds ($\xi$ and $\rho$) to improve detection at the expense of a slight increase in FP. Currently, they have been set automatically based on making 0.5% FP for the training data. Finally, 5% noise is almost at the high-end of the noise and thus for actual deployment where such noise level will be much lower, we should expect closer performance to that of the clean dataset.

• Performance on high-resolution dataset

  Ans:To get quick result we had to resort to two other moderate/high-resolution datasets, instead of the imagenet dataset that the reviewer suggested. We used Stanford car [2] dataset that contains images of size 400x400 pixels and the Youtube face (YTF) [3] dataset that contains images of size 100x100 pixels. Stanford-car consists of 8,144 train samples and 8,041 test samples and it has 196 classes. YTF consists of 1,15,470 training samples, 12,592 test sample and 1283 classes. The only adjustment we had to do is the use of larger triggers (8x8). As the below table shows, with this change our defense remained strong against such moderate/high resolution images. Table 2: Detection performance on datasets with high-resolution images.

  ---------------------------------------------------------------------------------------------------
  |     Dataset   | Clean Acc.|  eps  |  FP  | FGSM |  PGD  |  CW   |   DF  |  PA   | AA |
  |--------------------------------------------------------------------------------------------------
  | Stanford Car  |   97.45   | 4/255 | 0.8  |  100 | 99.98 | 96.43 | 97.62 | 99.07 | 100 | 98.22   |
  | Youtube Face  |   98.94   | 4/255 | 3.17 |  100 |  100  | 99.99 | 98.08 | 97.03 | 100 | 97.31   |
  ---------------------------------------------------------------------------------------------------
  

• Size of the extra dataset

  Ans: The extra datasets corresponding to robust and the backdoor loss functions can be created on the fly during the batch creation and thus we don't need to create separate datasets (more in section 4.1, and Table 4 in the supplementary). However, due to the effect of so many loss functions, we needed to train our models for longer time. For example, to achieve similar level of clean data accuracy on WideResNet (width-8, depth-20) we had to train our model on CIFAR-10 for 3,000 epochs, which is roughly 3-4 times that of learning a model only on the clean data. This is one of the significant requirement of our method.

[1] Tianyu Gu, Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. Badnets: Evaluating backdooring attacks on deep neural networks.

[2] https://www.cv-foundation.org/openaccess/content_iccv_workshops_2013/W19/papers/Krause_3D_Object_Representations_2013_ICCV_paper.pdf

[3] arXiv preprint arXiv:1712.05526, 2017.