Human-Producible Adversarial Examples

We thank the reviewer for their comments. We will address the reviewer’s points in order:

Lack of motivation. The reviewer is not convinced about why “human-producible adversarial examples” are desired in the first place, given that electronic devices, e.g., printers and scanners, are quite easy to get nowadays. More importantly, printed adversarial examples have been shown to be very robust in the literature and the proposed “human-producible adversarial examples” have not saved any computational cost at all. From the perspective of the budget of the attacker vs. defender, it seems not reasonable to constrain the attacker to use only pens since the defender has the ability to use a model/camera/scanner. More specifically, in this paper, the authors indeed printed out those adversarial images with lines and let humans reproduce them. In this case, why don’t the authors directly use printed images as the final adversarial examples? By the way, these printed adversarial images should be compared as a baseline attack.

The authors tried but did not find any baseline appropriate to the threat model that the paper is addressing. Overall, due to our proposed attack only requiring a marking pen or a roll of tape, it is more portable and has a high access factor compared to methods that require printers or carrying around suspicious prefabricated patches. We can in principle report the numbers from other papers side by side but find this to be apples to oranges and not really representative of overall attack usefulness.

Human-producible Adversarial Examples do not pursue cutting down computational costs, but rather focus on producing adversarial examples that humans can use without additional training or sophisticated equipment. It is not at all easy to manufacture adversarial glasses or an adversarial tshirt, but we find that it is trivial for humans to apply lines, making it a viable attack in a low-budget scenario where additional tools are not available.

Lack of discussion about potential countermeasures. A clear drawback of using a simple adversarial manipulation is the lack of stealthiness. In the case of line-based adversarial tags, it seems very easy to filter out them, e.g., based on edge detection or simply prototype matching. The authors should better test these potential countermeasures because attack stealthiness is important.

We thank the reviewer for pointing this out. We believe countermeasures such as edge detection or prototype-match would be very application-specific to be useful as lines occur often in the real world. In the same way as all of the other work, e.g. adversarial tshirts and glasses, we leave defences to future work.

As can be seen from the visualizations, the robust loss leads to lines quite close to the cup but the non-robust loss leads to lines further away. Why does it happen? If this really the case about why the robust loss works better, is it possible to just constrain the modification space to be surrounding the cup to make the attack robust?

We do not observe any common traits or patterns emerging from the robust and non-robust loss on similar images, and believe that the observations pertaining to lines closer to the subject leading to more robust loss are just coincidences. We believe that, much like for classic pixel-based adversarial example generation, there would be no human-discernable patterns that emerge.

The detailed image design was not introduced until Section 3.5 “To test this real-world scenario, we conducted an experiment whereby we took photographs of a common household object – a cup – and produced an adversarial tag for them. We constrained the search area to a rectangular bounding box to limit the lines to a specific area of the image to avoid the cup itself.” However, without this introduction, it is very hard to understand the specific design depicted in Figure 2. The reviewer initially thought there was a sub-figure showing a cup pasted in the main figure showing the wall. So the suggestion is, for Figure 2, to detail those two factors independently but leave out the messages about the image design. By the way, why doesn’t the black area occupy the whole image?

We acknowledge the confusing order of these two parts of the paper and now rephrase the sections. The explored black area is an artefact of the attack process line placement, as well as, jitter.

The use of “real world”/“physical world”. In the experiments, the authors consider drawing lines and pasting tapes. It is better to separate these two settings using more specific terms beyond the “real world”/ “physical world”.

We thank the reviewer for bringing this to our attention. We are not entirely sure what the reviewer sees as the difference between the settings as both drawing lines and pasting tape achieve the same results of introducing the black lines into the real world. Otherwise, we use the real and physical world interchangeably.

We thank the reviewer for their comments. We will address the reviewer’s points in order:

The most significant shortcoming of this paper is the extremely insufficient evaluations. Attacks like the ones proposed are designed to be executed in real-world scenarios, which are typically "black-box" in nature. To only assess such attacks in "white-box" settings is not adequate; after all, any attack designed to maximize the loss function might perform well against known classifiers in such a setting.
Additionally, the experimentation is conducted exclusively on the YOLOv8 model. To ensure a thorough understanding of the attack's efficacy, it's crucial to extend the evaluations to include a broader range of models.

We acknowledge the limited evaluation in the paper. We intended the paper to be a capability one. We now include black box transferability results in Appendix in Tables 2 and 3, but also would like to justify our choice of white-box evaluation on YoloV8. YoloV8 and the yolo family of models is one of the most widely-deployed ML models for vision and is often used out-of-the-box. Hence, we feel that the results of the white-box evaluation fit the realistic physical-world threat model outlined in the paper.

In real-life conditions, adversarial implementations may face various distortions due to camera angles or environmental interferences like weather. However, the paper's evaluation seems too idealistic. The authors have chosen an optimal camera angle, and there is a noticeable absence of corruption in the adversarial tags. This approach doesn't fully simulate the real-world conditions where such adversarial techniques would be applied.

We address the potential corruption and transformation misalignments that may come about due to improper recreation of the lines by introducing the jitter and erasure factors in the robust loss calculation. We refer the reviewer to Section 3.2 and Figure 2 that explains it in detail.

As shown in the user study, these significantly improve the real-world efficacy of the attack. As we note in the paper, in practice other corruptions can be introduced alongside to gain invariance of choice e.g. lighting or angles.

We thank the reviewer for their comments. We will address the reviewer’s points in order:

The "Method" section would benefit from additional granularity. Specifically, it remains unclear how overlapping of the randomly generated lines is addressed. Are there any constraints or specific guidelines governing the generation of these random lines?

We did not enforce particular constraints such as non-overlapping on the generated lines, as overlapping lines are still replicable in the real world. The generation of lines includes two stages. First, we sample adversarial lines uniformly at random over the full extent of the image (or a rectangular subset if specified). Second, we iteratively adjust the position of these lines via gradient descent. The gradient is calculated directly on the rasterized lines following "Differentiable Drawing and Sketching" (Daniela Mihai and Jonathon Hare, 2021).

The decision to employ only four line-defining points warrants clarification. Is there a theoretical foundation or empirical rationale that supports this choice?

We outline in section 4.1 that the method can be applied to Bezier curves and more complicated shapes, but we restrict our investigation to straight lines, which can be uniquely defined by their start and end coordinates, which is where the 4 parameters come from.

There are noticeable writing inconsistencies. For instance, the abbreviation "NLL" is employed prior to its formal definition, which might be confusing for readers.

We apologise for the inconsistencies and many thanks for pointing this out! It is now corrected in the manuscript.

What underlying principles or mechanisms allow simple lines to effectively execute adversarial attacks?
When considering both targeted and untargeted attacks on images of similar objects, what common traits or patterns emerge?

Adversarial examples still puzzle researchers 10 years after their discovery; we also, unfortunately, still do not have a solution to them. In our work we highlight that something as simple as a few lines can be enough to disrupt such a sophisticated model as YOLO; importantly, with our work humans can trick YOLO effectively.

We follow the attack process that is common in the adversarial example literature. We explicitly search for adversarial perturbation that maximises the classification loss within a space represented by lines. The attack’s success suggests that the classic pixel-based adversarial perturbation also exists in a far simpler space represented by lines. Our findings are also consistent with existing adversarial examples literature in that we do not observe any common traits or patterns emerging from targeted and untargeted attacks on similar images.

The motivation presented could be better articulated. What is the rationale behind necessitating human production of the adversarial line? Is there an inherent advantage or specific scenario where this becomes crucial?

We acknowledge the reviewer’s concerns regarding the motivation of the paper.

Attacks are categorised by their costs. Usually more sophisticated attacks come with higher costs, making them harder to use in practice. We see our work as a natural continuation of physical adversarial examples literature, where we now enable faster (e.g. require application of a few lines, humans manage it in seconds), deniable (e.g. its just a few lines outside of the object, no odd looking adversarial patches, an attacker doesn't need to carry around suspicious looking patches), cheaper (e.g. drawing pens are available everywhere, graffiti pens are used all across the world), and usable attacks (e.g. the first attack of its kinda that is easily applicable by human, as is proven by our human evaluation).

In scenarios where the lines are confined solely within the object's area, does the efficacy of the proposed attack remain consistent?

We find that the attack remains successful when confining the lines within the object’s area. Additionally, we clarify that constraining to the object area is an “easier” attack, as it can easily change the existing object’s appearance to fool the classifier. Our setting is arguably more challenging because we need to change the model’s output without perturbing the object itself, but the background around it.

We thank the reviewer for their comments. We will address the reviewer’s points in order:

The user study consisted of only four participants, and does not investigate the effects of artistic ability. The sample size seems too low to draw any broad conclusions. It wasn't mentioned if the participants had a limited time budget to replicate the lines, which might be an important consideration in replication.
In the human study, how much time were participants given to replicate the lines?

We have subsequently recruited 10 additional users with the same setting and observed consistent results. Throughout the user study, we did not enforce a time budget and asked the participants to replicate lines as flexibly as they liked. The timings are now presented in Appendix D of the paper, and show that even for 12 lines, users were able to create the adversarial tag in under 20 seconds. We did not investigate artistic ability because our focus is to evaluate the robustness of replication, and having artistic ability would only make the replication more precise which goes against our intention to evaluate the "bad" replication cases. Results are updated in the paper.

The evaluation only considers a single YoloV8 classifier, rather than checking on multiple architectures and robustness levels, which are valid in the author's threat model. I was expecting experiments on robustified models [1,2] or some experimental results in trying to use adversarial lines for adversarial training. It would be interesting to see if the data generated by adversarial lines is too noisy, potentially degrading the benign accuracy when used for AT. This could provide useful information for the broader community.
How well do robust models fare against the adversarial lines? Can adversarial lines be used for adversarial training?

First, we want to note that robustness literature referred to is either rarely used in practice because of latency overheads e.g. with randomised smoothing, or incurs significant accuracy degradation e.g. with adversarial training. Second, even if we assume that adversarial training or smoothing is employed, the perturbation budget of the lines is significantly larger than anything that certification in current literature may provide. We now include Table 2 and 3 in Appendix B that show that we utilise perturbation with Linf norms from 214/255 to 242/255, compared to common certification of 8/255.

Only the white-box implementation is investigated, so the transferability of lines is unknown. It seems unlikely that a human with only a marker can also backprop through a service provider's model. I was expecting an experiment where the adversary tries to transfer adversarial lines from an owned model to an unseen model, potentially of a different architecture. The authors primarily pitch the contribution as an easily accessible technique for non-experts, so it seemed contradictory that they would also require access to weight-level knowledge of the defender's model. In the same spirit, the computational complexity seems too high for a non-expert to perform these attacks, since it requires a 4-GPU workstation to run.
Can the authors comment on the attack transferability? Is it feasible that adversarial lines would transfer to an unseen model?

We acknowledge the limited evaluation in the paper. We intended the paper to be a capability one. We now added the black box transferability results to Table 4 in Appendix C – we do find evidence that adversarial examples transfer across the yolo family.

We next would like to justify our choice of white-box evaluation on YoloV8. YoloV8 and the yolo family of models is one of the most widely-deployed ML models for vision and is often used out-of-the-box. Hence, we feel that the results of the white-box evaluation fit the realistic physical-world threat model outlined in the paper.

We clarify that “capability” refers to the user’s capability of realizing a given adversarial perturbation. Hence, the computational cost for computing the perturbation is outside the scope of a user’s capability, e.g., by deploying the attack on an easy-to-access cloud server. We consider the retrieval and realization of adversarial perturbation to be two disentangled stages. At the same time, the method itself is not particularly memory or GPU-heavy and could likely be run on a modern smartphone, albeit with a larger time budget.