Can Adversarial Examples Be Parsed to Reveal Victim Model Information?

• threat model is not clear
• the real challenge is to recognize unseen models
• no discussion on the limitations
• not tested on robust models
• poor visualization of figures and tables

High-Level

• The "practical" usage requires to detect the adversarial examples first (which is far from trivial)
• Missing "intermediate" experiment on the detection of AE
• Only approachees the problem from a computer vision perspective (and only CNN)
• Some (fixable) terminology issues
• Unclear if the code will be disclosed
• Some parts are hard to interpret (and poor choice of OOD evaluation)

Comment

Even though the paper has arguably many strengths (indeed, I was not able to point a clear "flaw" in the paper), I do have some concerns w.r.t. its significance and its overall contribution to the state-of-the-art. Indeed, while I do appreciate the novelty, I believe that the "practical value" of these findings to be low. In other words: how can the real-world benefit from this paper? On the one hand, I feel that the paper lacks a clear statement of how the proposed findings can be leveraged by future work; on the other hand, I feel that --to be usable-- these findings require some intermediate step (i.e., detecting AE) which is difficult to realise in practice.

Still, all such issues are fixable (to some extent) but some extra work is required. In what follows, I will provide detailed explanations and potential ways to address all the weaknesses outlined above.

Practical usage

To my understanding, the only way to truly use the findings of this paper "in practice" is by detecting an adversarial example first, and then using it as basis to infer information on the corresponding VM. However, doing this is far from simple, as evidenced by many recent works (e.g., [F]). Hence, I wonder: what other usage is there for the method proposed in this paper? The text does mention that it opens new possibilities in the field of "robustness". To quote specific statements:

If feasible, the insight into model parsing could offer us an in-depth understanding of the encountered threat model and inspire new designs of adversarial defenses and robust models.

The insight into model parsing could offer us an in-depth understanding of the encountered threat model and inspire new designs of adversarial defenses and robust models.

...however, I question: how could this be done in practice? The paper does not elucidate how these insights can be leveraged for this purpose -- and, imho, I do not see any way to do so. For instance, from the viewpoint of a researcher (who is well-aware of the experimental pipeline, and hence "knows" what adversarial example is used to attack a given model), I do not see any way to leverage this. I invite the authors to discuss this---potentially by reducing the space given to the discussion of the results (which, personally, I found to be quite cumbersome to interpret).

Missing "intermediate" experiment on the detection of AE ("How many samples are needed?")

Tieing back to the previous "open question", I am wondering how many samples are necessary "in practice" to understand that an attacker "truly" used a given VM. To my understanding, the process should be as follows: assume that an attacker, A, wants to evade a model M by means of some adversarial example x'. The defender is adopting a countermeasure, D, to detect adversarial examples. The attacker crafts the adversarial example x' by creating a surrogate of M (i.e., VM) and then transferring the adversarial examples x' to M. Let's call X' the set of all adversarial examples that bypassed VM and that are used to attack M.

In the experiments, the assumption is that X' is used to "test" (or "train") the proposed MPN. However, in reality, only a subset of X' would be detected by D (given that detection mechanisms are not perfect). Let's call S such a subset. The defender can then do the "forensic" analysis only on S (and not on X').

My question is: what is the performance of MPN on S? And, more importantly, how large is such S? Because if S only has, say, 10 samples (whereas X' has 1000) then it may be unpractical to derive any sound conclusion from the accuracy of an MPN tested on S.

My stance is that, from a practical viewpoint, it would be insightful to perform an additional analysis focused on determining the feasibility of the approach by assuming an "adversarial detection method" used to collect the samples that will then be further analysed. This would resemble a realistic scenario that allows to shed light on the attacker tactics. Perhaps a discussion of such a "practical use-case" could be added in an appendix (I would love to read something like this!)

Only image data (and only Deep Learning)

The paper only considers ML applications within the deep learning (DL) paradigm, and -- specifically -- onl Convolutional Neural Networks that analysed Image data. However, as shown by a plethora of papers, the breadth of domains in which ML has found applications (and which can hence be attacked via adversarial examples) is much broader. Such a limited scope is, in my opinion, a limitation of this analysis: I would be delighted to see the proposed mechanism be applied in a different context, potentially entalining ML models that have nothing to do with neural networks and/or which analyse a different data type (e.g., time series; or tabular data).

Some (fixable) terminology issues

The paper consistently relies on terminology that has been recently questioned [A].

For instance, I invite the authors to revise the usage of "black/white-box" and use "zero/perfect knowledge" instead (similarly, WB and BB should become PK and ZK).

Furthermore, the paper many times mentions the term adversarial attacks. According to [A], this is a thautology that is confusing to many. I endorse the authors to revise any occurrence of such term.

Moreover, there are a series of statements that I invite the authors to revise. In what follows, I will directly quote such statements and express my concerns.

have emerged as a primary security concern of ML in a wide range of vision applications

I disagree. Actually, there is evidence suggesting that adversarial ML attacks are not among the priorities of practitioners see [A,B,C,D]. At most, they are now seen as a means to achieve "robustness" rather than "security": these two goals, while similar, are orthogonal (even a very recent work mentioned "false state of robustness" [E])

The adversarial attack (a.k.a, adversarial example)

I disagree with such wording. An adversarial example is "not" an attack. The "attack" is the process of crafting the adversarial example with the specific purpose of subverting an ML model. Potentially, an adversarial attack entails the creation of multiple adversarial examples. Hence I endorse the authors to revise this association, which is imprecise from a security perspective (see [A]).

with full access to $\theta$

What does "full access" means? I endorse the authors to look into the "terminology issues" discussed in [A].

Finally, I must note that the caption of figures reports "Figure", but in the text it is referenced as "Fig."

Some parts are hard to interpret (and poor choice of OOD evaluation)

This is just a personal comment, but I felt that the presentation of the results and of the experimental methodology to be convoluted. For instance, I still did not fully understand how the OOD experiments have been carried out: taking Figure A1 as an example, its caption reports

performance of MPN when trained on a row-specific attack type, but evaluated on a column-specific attack type

Let's take the cell in the first row (PGD-l_inf) and second column (FGSM). Does it mean that the MPN is trained on PGD, and tested on FGSM? If so, does it mean that 66.3% of the times the MPN was able to infer the correct VM?

At the same time, the description in the main text is also vague:

In addition to new test-time images, there could exist attack/model distribution shifts in $\mathcal{D}test$ due to using new attack methods or model architectures, leading to unseen attack methods ($\mathcal{A}$) and victim models ($\Theta$) different from the settings in $\mathcal{D}_{tr}$.

While I do see some reason in assuming such a OOD setting (because, in reality, a defender does not know the spectrum of attack techniques used to craft the AE), there is also another one which is more interesting to consider: the case in which the OOD entails VM that are not known a-priori.

For instance, what would happen if the attacker crafted the AE using a CNN with a different architecture than the ones envisioned in Table 2? I feel that not investigating this case is a limitation from a practical perspective, given that the true goal of the "AE parsing" should be to infer the VM---but the space of potential VM is infinite, meaning that a 100% match is unfeasible to achieve in practice. Hence, the point is (IMHO!) assessing "how close to the real VM a given prediction is). For instance, assume that the attacker used ResNet18 but that ResNet18 is not included among the "classes" used to train the MPN: I would expect the MPN to predict "ResNet20" (since, among the remaining VM, it is the most similar); however, if the MPN predicts "VGG13", then the performance is "bad".

My stance is that the paper makes a "closed world assumption", which is unrealistic in practice. This, alongside the other weaknesses mentioned above, raises some skepticism on the overall "value" of this methodology (which, I stress, is novel and potentially interesting to some researchers!).

EXTERNAL REFERENCES

[A]: Apruzzese, Giovanni, et al. "“Real Attackers Don't Compute Gradients”: Bridging the Gap Between Adversarial ML Research and Practice." 2023 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML). IEEE, 2023.

[B]: Boenisch, Franziska, et al. "“I Never Thought About Securing My Machine Learning Systems”: A Study of Security and Privacy Awareness of Machine Learning Practitioners." Proceedings of Mensch und Computer 2021. 2021. 520-546.

[C]: Kumar, Ram Shankar Siva, et al. "Adversarial machine learning-industry perspectives." 2020 IEEE Security and Privacy Workshops (SPW). IEEE, 2020.

[D]: Bieringer, Lukas, et al. "Industrial practitioners' mental models of adversarial machine learning." Eighteenth Symposium on Usable Privacy and Security (SOUPS 2022). 2022.

[E]: Pintor, Maura, et al. "Indicators of attack failure: Debugging and improving optimization of adversarial examples." Advances in Neural Information Processing Systems 35 (2022): 23063-23076.

[F]: Tramer, Florian. "Detecting adversarial examples is (nearly) as hard as classifying them." International Conference on Machine Learning. PMLR, 2022.

• I'm unfamiliar to the research field of model parsing. I wonder if the authors can provide some comparisons to existing works that are closely related to model parsing?

1. The question of whether a strong correlation actually exists such that one can effectively a classification function that can predict a victim model attributes given adversarial instances is perhaps the elephant-in-the-room. Showcasing that a developed neural network is able to predict with $x$ accuracy the model parameters empirically and showing that the attack development optimization can involve the model parameter $\theta$ is a weak motivation.
2. Transferability of attacks is well-known in literature [1] and has been a major point in the design of defense mechanisms that use multiple victim models [2,3]. This sort of hints that the function form an attack to VM parameters is a one to many task. Yet, the classification task is designed as a one-to-one function.
3. One wonders what happens in the context of Universal Perturbations [4]? While these were originally designed to be a single perturbation vectors that can be used across the test set, what happens when you consider a generalized version that is effective across model architectures [5]?
4. In the black-box setting, what happen when the distilled model has a different architecture & VM attributes than the parent/target model? Precisely, will the learned attack vector showcase characteristics of the distilled model or the target model?
5. The results with the PEN network estimates $\delta_{PEN}$ seems quiet worse-off compared to using the perturbation $\delta$ (eg. 62.2 vs. 99.66 for PGD $l_2, \epsilon=0.25$ on ResNet9, 66.38 vs. 91.84 for $l_\infty, \epsilon=4/255$ on VGG 11, etc.) Is there a good analysis on why this is these cases? Is it a problem with the $PEN$ network? Why is the disparity so high for small perturbations like $\epsilon=4/255, 0.25$ (where I was expecting higher probability of unique signatures for model types) and much less for higher perturbations like $\epsilon=16/255, 1$?

[1] Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., & Fergus, R. (2013). Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199.

[2] Sengupta, S., Chakraborti, T., & Kambhampati, S. (2019). Mtdeep: boosting the security of deep neural nets against adversarial attacks with moving target defense. In Decision and Game Theory for Security: 10th International Conference, GameSec 2019, Stockholm, Sweden, October 30–November 1, 2019, Proceedings 10 (pp. 479-491). Springer International Publishing.

[3] Adam, G. A., Smirnov, P., Duvenaud, D., Haibe-Kains, B., & Goldenberg, A. (2018). Stochastic combinatorial ensembles for defending against adversarial examples. arXiv preprint arXiv:1808.06645.

[4] Moosavi-Dezfooli, S. M., Fawzi, A., Fawzi, O., & Frossard, P. (2017). Universal adversarial perturbations. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1765-1773).

[5] Chaubey, A., Agrawal, N., Barnwal, K., Guliani, K. K., & Mehta, P. (2020). Universal adversarial perturbations: A survey. arXiv preprint arXiv:2005.08087.

• The threat model does not make much sense to me. For instance, if the attacker uses an attack like PGD to construct adversarial examples, it has to have white-box access to the model anyway. Then what is the point of having inference for attributes of the model that are anyway known? Such inference would only make sense for black-box attacks.

• Table 1: The selection of attacks seems arbitrary. They are not state-of-the-art (there are dozens of attacks that are better, such as MIDIFGSM. See this survey for an extensive analysis). Also, for these attacks and their hyper-parameters, please also mention the number of iterations the attacks are run for, etc.

• Section 4: "We first create a model parsing dataset by collecting adversarial attack instances against victim models." I am confused about the setup- you're using adversarial examples generated against victim models, to learn their attributes? Shouldn't the models at test time be different from the ones you used in training? Assuming these models are indeed trained (or, at least, available), a simple approach I can think of is to just evaluate all of them (on some data sample X) and pick the one that seems to give the best performance on target model. Why go through this hassle?

• Section 5: "The generalization is measured by testing accuracy averaged over attribute-wise predictions, namely..." Odd metric- is this standard? Why is the number of classes part of the metric? Also, should ideally report attribute-wise classification accuracies as well.

• Pg9: "PN to uncover real VM attributes of transfer attacks." - So the results before used the same model as the one used in training the attribute-inference model? No wonder it works so well - this experiment right here should be the main focus of analysis.

Minor Issues

• Pg 3 (and other references): 'VM' is an unnecessarily confusing abbreviation. Please use 'victim model' or 'target model'
• Pg 3, RED: Seems to be a very new and niche concept, and readers should not be expected to know about it much- please explain how this is useful, what can a victim do if it knows adversary is running untargeted adv attack, etc.
• Pg 5, "(Problem statement) Is it possible to infer victim model information from adversarial attacks? And what factors will influence such model parsing ability?" I can reason about the motivation for this (knowledge could allow using closer local models, etc.), but should be stated more explicitly in the paper to motivate the threat model. One concrete way to do this would be: assume attribute inference works perfectly, how much of a jump in attack success can it get? Compare it with an attack that has the same query/compute budget, and see if this approach works better.
• Pg 6, "thus adversarial attacks in Dtest are new to Dtr" - How does that matter? It should be the models that are new.
• Table 4: Looking at results across makes me think the choice of adv attack will not matter much, but should include experiments nonetheless to confirm.
• Figure 7: Way too small text in heatmap- okay to only include representative ones (and leave bigger version for Appendix) but please make text labels bigger