On the Vulnerability of Adversarially Trained Models Against Two-faced Attacks

Thank you for your precious time in reviewing our work. Our point-by-point responses to the reviewer's mentioned questions (Q) and weaknesses (W) are provided as follows.

Q1: Can the authors provide a real-world example that the two-faced attacks can be applied into?

According to the statistics in practical deployment [1], 85% of attempted deployments ultimately fail to achieve the intended results in the production environment. Validation is one of the keys to a successful deployment [2]. A practical real-world example is computer virus detection. There is a possibility that the collected verification data used to validate the virus detection software has been already influenced by attackers using two-faced attacks, which would lead to falsely elevated verification results. In this case, the deployment of such a virus detection software may result in serious consequences.

[1] https://web.archive.org/web/20220707220351/https://www.decisioniq.com/blog/ai-project-failure-rates-are-high

[2] Andrei Paleyes, Raoul-Gabriel Urma, and Neil D. Lawrence. Challenges in deploying machine learning: A survey of case studies. ACM Computer Survey, 55(6), 2022.

Q2: Can the authors provide more potential countermeasures against two-faced attacks, apart from the ones mentioned in Section 3.3?

In the submission, the main focus is on revealing the threat of two-faced attacks for the first time, while a mitigation strategy by increasing the defense budget was presented. In addition, we experiment with THRM [1] as another mitigation strategy. Specifically, we train ResNet-18 with THRM using different $\lambda$ under the $\ell_{\infty}$ norm and $\epsilon=8/255$ on CIFAR-10 and CIFAR-100. The experimental results are as follows:

1. The experiment of THRM with different $\lambda$ under $\ell_{\infty}$ norm and $\epsilon=8/255$ against two-faced attacks on CIFAR-10:

2. The experiment of THRM with different $\lambda$ under $\ell_{\infty}$ norm and $\epsilon=8/255$ against two-faced attacks on CIFAR-100:

It can be seen from the above experimental results that the aforementioned strategies partially mitigate the risk of two-faced attacks, but such a risk remains significant and cannot be ignored. We have updated these experimental results in the appendix.

Furthermore, we can incorporate our two-faced examples (derived from the two-faced risk) into existing adversarial training frameworks, such as TRADES and MART. In these frameworks, KL divergence is used to measure the difference in the output distributions between clean and adversarial examples. We can simply replace the adversarial examples in the KL loss term with our two-faced examples.

However, because of the iterative generation of two-faced examples in a bi-level optimization formulation, the whole optimization process (i.e., alternately generating two-faced examples and conducting adversarial training) is highly time-consuming. Hence, we did not adopt this countermeasure. We will explore potential methods to improve the optimization efficiency for incorporating our generation of two-faced examples into existing adversarial training frameworks, in future work.

[1] Lue Tao, Lei Feng, Jinfeng Yi, and Songcan Chen. With false friends like these, who can notice mistakes? In AAAI, 2022b.

Continue by the next post.

W1: The two-faced attacks are the key of this paper, but the realistic application domains are still unclear.

Two-faced attacks focus on the robustness during the model verification phase. The verification phase is significant, especially in security-critical applications such as autonomous driving or computer virus detection. Additionally, considering the need for long-term monitoring and updates during the extended deployment of models, two-faced attacks could be applicable in this phase as well. Due to the high cost of detection and update, researchers may be deceived not to update the problematic model in the face of long-term false high validation results, which may eventually lead to a series of security problems.

We hope that our responses can satisfactorily address your concerns. Thank you again for your time to provide us with valuable feedback.

W3: The contribution of this paper is limited because the content is very similar to the previous work (Tao, 2022b), but changes from clean test samples to adversarial test samples. The authors may focus more on the countermeasure.

As you mentioned, the previous work (Tao, 2022b) focused on accuracy, while our work focuses on robustness during the verification phase. Hence there is a clear difference, which makes them two entirely different tasks. Our contributions to the task of two-faced attacks are completely new. In addition, we have made some new findings regarding two-faced attacks:

1. Models with different architectures and adversarial training methods are susceptible to two-faced attacks, leading to falsely elevated verification robustness;
2. Models with higher adversarial robustness tend to be more vulnerable to two-faced attacks.

Therefore, two-faced attacks are pervasive and tricky, and cannot be simply ignored.

In terms of countermeasures, we have included more experimental results:

1. Experimental results of PGD-AT with different budgets under $\ell_{\infty}$ norm on CIFAR-100;
2. Experimental results of THRM with different $\lambda$ under $\ell_{\infty}$ norm on CIFAR-10;
3. Experimental results of THRM with different $\lambda$ under $\ell_{\infty}$ norm on CIFAR-100.

We presented the relevant results in W2 and have added them to the appendix of the paper.

We hope that our responses and the updated experiments have addressed your concerns. We are always willing to discuss and address any further questions.

We thank the reviewer for the valuable and constructive comments, which provide considerably helpful guidance to improve the quality of our paper. Our point-by-point responses to the reviewer's mentioned questions (Q) and weaknesses (W) are provided as follows.

Q1: In Appendix A, I do not know how you derive from E[1(f(x'') = y) * 1(f(x_{adv} \neq y)] to E[1(f_{adv} \neq y)] * E[1(f(x'') = y)]. It is in the lines 7 and 8 if you count the line R_{rhpy}(f,D) as line 1.

Thanks so much for raising this point, which helped us identify an oversight in our derivation process and avoid the omission of a component in the final result. Fortunately, this has no impact on our conclusions. We have made corrections and uploaded the revised version. Thanks again for the reviewer's carefulness!

W1: The background and motivation are not convincing to me.

We have further enriched our background and motivation, and highlighted these changes in the introduction section of the paper. These can be summarized as follows.

Due to the high probability of failures of model deployment in practical applications, it is necessary for practitioners to conduct pre-deployment verification and post-deployment long-term verification to enhance the success rate of deployment [1], especially in safety-critical tasks. Therefore, this work explores the existence of verification-time threats that hypocritically fool the robustly trained models by slightly perturbing the verification data.

[1] Andrei Paleyes, Raoul-Gabriel Urma, and Neil D. Lawrence. Challenges in deploying machine learning: A survey of case studies. ACM Computer Survey, 55(6), 2022.

W2: The experiments for the countermeasure (enlarging the budget) are limited. The authors may need to add more experiments for it.

Thanks for the suggestion. We have conducted additional experiments to supplement the experimental results.

1. We trained ResNet-18 on CIFAR-100, using PGD-AT with different budgets under $\ell_{\infty}$ norm:

2. We trained ResNet-18 on CIFAR-10, using THRM [1] with different $\lambda$ under $\ell_{\infty}$ norm:

3. We trained ResNet-18 on CIFAR-100, using THRM with different $\lambda$ under $\ell_{\infty}$ norm:

[1] Lue Tao, Lei Feng, Jinfeng Yi, and Songcan Chen. With false friends like these, who can notice mistakes? In AAAI, 2022b.

Continue by the next post.

Dear Reviewer ox78,

Sorry to disturb you. We sincerely appreciate your valuable comments. We understand that you may be too busy to check our rebuttal. We would like to further provide brief answers here to the issues that may be your primary concerns.

We appreciate your inquiries regarding the formula derivations, which helped us to identify an oversight in our derivation process. Fortunately, this has no impact on our conclusions.

In addition, for the countermeasures, we have supplemented more experiments, including experiments on different datasets and experiments on the THRM defense strategy. We also discussed the possibility of incorporating two-faced samples into adversarial training methods as countermeasures.

Finally, our contributions significantly differ from the previous work (Tao, 2022b) that focused on the accuracy. We concentrate on the robustness in the verification phase. Our contributions to the task of two-faced attacks are completely new. In addition, we have made some new findings regarding two-faced attacks:

1. Models with different architectures and adversarial training methods are susceptible to two-faced attacks, leading to falsely elevated verification robustness;
2. Models with higher adversarial robustness tend to be more vulnerable to two-faced attacks.

We sincerely hope that the above answers can address your concerns. We look forward to your response and are willing to answer any questions.

W2: May not provide enough information about mitigation strategies.

In this paper, our primary focus is for the first time to reveal the threat of two-faced attacks. Regarding mitigation strategies, in addition to increasing the budget as mentioned in the paper, we also explored the mitigation capability of THRM [1] with different $\lambda$ under $\ell_{\infty}$ norm and $\epsilon=8/255$ against two-faced attacks on CIFAR-10:

The experiment of THRM with different $\lambda$ under $\ell_{\infty}$ norm and $\epsilon=8/255$ against two-faced attacks on CIFAR-100:

For the experiment of increasing the budget for PGD-AT, we further extended it to the CIFAR-100 dataset:

From the above experimental results, it can be observed that the aforementioned strategies partially mitigate the impact of two-faced attacks, but such a risk remains significant and cannot be ignored. We have updated these experimental results in the appendix.

In addition, for other potential mitigation strategies, as mentioned in Q1, we can integrate $x_{\text{tf}}$ into certain existing adversarial training frameworks.

[1] Lue Tao, Lei Feng, Jinfeng Yi, and Songcan Chen. With false friends like these, who can notice mistakes? In AAAI, 2022b.

Continue by the next post.

W3: There may be an overemphasis on two-faced attacks, which might mask other important security issues that need to be taken seriously.

We agree with the reviewer's perspective. The primary research objective of this paper is to uncover the threats posed by two-faced attacks, but it is undeniable that other significant security issues should not be overlooked. Balancing the risks associated with two-faced attacks and other security concerns is indeed a topic worthy of in-depth exploration. For instance, integrating the two-faced risk into existing adversarial training frameworks merits thorough investigation. However, as discussed in the response to Q1, the resulting optimization process (i.e., alternately generating two-faced examples and conducting adversarial training) is highly time-consuming, especially given vast training data. Therefore, there is a need for efficiency optimization to address this issue.

W4: A lack of clarity in the procedures for experiments may make them difficult to repeat and inhibit future studies.

Thanks for raising this concern. We have provided more information in the experimental procedure section in Appendix B and have uploaded the revised version.

W5: There can be a gap in comprehensive risk management techniques if the paper does not include a comprehensive explanation of how to balance adversarial risks, such as two-faced risk, against other risks.

We agree with the reviewer's feedback. The issue of balancing these risks is indeed worth investigating. While our primary focus in this paper is to uncover the threats posed by two-faced attacks, we presented a preliminary exploration of the trade-off between adversarial risk and two-faced risk in Figure 3 of the submission. As mentioned in the response to Q1, the resulting optimization process is computationally inefficient. Hence, we plan to delve into these challenges in future work, and we anticipate that the reviewer understands the complexities involved.

We hope that our responses can satisfactorily address your concerns. Thank you again for your time to provide us with valuable feedback.

Thanks for the valuable and encouraging comments! We are so grateful to see that our carefully prepared work can be recognized by the reviewer. Our point-by-point responses to the reviewer's mentioned questions (Q) and weaknesses (W) are provided as follows.

Q1: Is it possible to include the idea of two-faced risk into current adversarial training frameworks without making major changes?

Yes, we can incorporate our two-faced examples (derived from the two-faced risk) into existing adversarial training frameworks, such as TRADES and MART. In these algorithms, KL divergence is used to measure the difference in output distributions between clean and adversarial examples. We can simply replace the adversarial examples in the KL divergence term with our two-faced examples.

However, because of the iterative generation of two-faced examples in a bi-level optimization formulation, the whole optimization process (i.e., alternately generating two-faced examples and conducting adversarial training) is highly time-consuming. Hence, we did not adopt this countermeasure. We will explore potential methods to improve the optimization efficiency for incorporating our generation of two-faced examples into existing adversarial training frameworks, in future work.

Q2: How can long-term model validation procedures be affected by the future evolution of two-faced attacks?

If the attacker persists in conducting two-faced attacks, the long-term model verification procedures may also exhibit falsely elevated performance. Since long-term validation and model updates are challenging and time-consuming, researchers may not update their models if they observe falsely elevated validation metrics for a long time, opting instead to continue deploying a potentially problematic model. This could ultimately lead to a cascade of security issues. This is also why we emphasize in the paper not to overly trust these validation results.

W1: Though interesting, its theoretical investigation of two-faced risk may prove difficult to implement in real-world scenarios.

We would like to clarify that, for applications in the real digital world, our two-faced attacks are easily applicable. As long as researchers collect verification data from the online world, there is a risk of being vulnerable to two-faced attacks. For applications in the physical world, e.g., autonomous driving, two-faced attacks are feasible by combining them with techniques such as adversarial patches. Overall, two-faced attacks are a real threat to AI if adversaries are aware of where the validation data is collected.

Continue by the next post.

Thank you for your interest in our work!

We would like to clarify that $-\delta^{\star}$ may not necessarily be the optimal solution that satisfies the formulation. In Algorithm 1 of our paper, we show the optimization process for $x_{\text{tf}}$. In Line 3, when calculating the optimal value $t^{\star}$, due to the nonlinearity of neural networks, there may exist a solution that results in a smaller loss than $-\delta^{\star}$. The same rationale also holds for Line 4, when calculating the optimal value $\delta^{\star}$.

Besides, the two-faced risk on the non-robust dataset ($D_{f_{\theta}}^{\text{cnr}}$) is not always equal to 1. When conducting adversarial attacks on $x_{\text{tf}}$ generated from $D_{f_{\theta}}^{\text{cnr}}$, the attack algorithm may not produce a countervailing perturbation (i.e., make $x_{\text{tf}}$ recover the clean $\boldsymbol{x}$ after adding an adversarial perturbation). This is because such a perturbation might not optimally lead to the misclassification. Due to the nonlinearity of neural networks, there may exist perturbations in other directions that lead to the misclassification of the perturbed $x_{\text{tf}}$. Therefore, the two-faced risk on $D_{f_{\theta}}^{\text{cnr}}$ is potentially less than $1$.

We hope that our response can address your concerns. Thanks again for your attention to our work and for your comment.