PADetBench: Towards Benchmarking Physical Attacks against Object Detection

1. The presentation can be improved. Although comprehensive experimental results are necessary, some crowd the layout and cause a terrible experience. For instance, Figure 2 provides extensive results while each sub-figure holds only a small subset of space. It makes readers try their best to broaden the figure and keep their eyes fixed on it. Selectively displaying results may be a better choice.
2. The comparison of other benchmarks in object detection should be available. Since the benchmark is proposed to illustrate the data, detectors, and adversarial attacks in object detection, the comparison of previous works or demonstration of the applied methods is important. In fact, the corresponding information is stated in Table 1 (target objects) and Table 2 (detectors). However, a clear comparison of the previous benchmarks is missing, which can be introduced in a list like the tables above.
3. The applied objects and physical simulator might be limited. Various detectors and adversarial attacks are utilized in the paper, while the physical simulator and the applied objects are limited. For example, more objects like border trees besides vehicles, persons, and traffic signs can be considered. Furthermore, the fixed simulator is likely to retain a special pattern in the generated data, which may lead to less reliable results on the benchmark.
4. The transferability of adversarial attacks should be considered in the analysis. In practice, the detectors are unknown to attackers, usually named as a "black-box" setting in adversarial attacks, and attackers are likely to create adversarial examples based on a substitute model by the transferability of adversarial examples. The analysis of the transferability of adversarial examples in object detection seems absent, though the evaluation in white-box settings is available (e.g., Figure 3 and Figure 4).

I appreciate that the authors present extensive experimental results to assess the performance of physical adversarial attacks and address the shortcomings of current benchmarks, including their time-consuming and costly nature, challenges in aligning physical dynamics, cross-domain loss, and difficulties in comparison (lines 46-53). However, readers unfamiliar with physical adversarial attacks may struggle to understand the significance of these issues and how the proposed benchmark effectively addresses them. As it currently stands, the paper resembles a shopping list, making it difficult for readers to grasp how the problem is solved amidst the multitude of experimental results.

Specifically, the authors do not provide a quantitative metric to demonstrate that the previous benchmark is time-consuming or how the proposed benchmark mitigates this issue. The same critique applies to the other three points. Additionally, adversarial attacks on object detection encompass at least five objectives: appearing attack, hiding attacks, mis-classifying attacks, mis-locating attacks, and latency attacks [1]. While not all objectives have been implemented in physical attacks, the authors should clarify the scope of the proposed metrics (Equations 1 and 2).

Furthermore, the corollaries presented (lines 423-460) do not appear to be novel; similar ideas have been explored in existing literature. The authors should provide stronger empirical evidence to demonstrate that the proposed benchmark is superior to existing works. Overall, I believe this work is a significant milestone for assessing the performance of physical adversarial attacks, but the writing style requires refinement, and the evidence supporting its superiority should be emphasized.

[1] Overload: Latency attacks on object detection for edge devices.

Despite creating a standard benchmark to compare what may be the performance of physical attacks in real-world scenarios, this paper falls short of creating such a benchmark. Too many details are missing, and the presentation could be significantly improved. Please find bellow the different weaknesses that I find necessary to address:

• A lack of comparison with other benchmarks of the literature. For example, Zhang et al. (2023b) generated the DCI dataset using the CARLA simulator. What is the difference between the author's work and the work of Zhang et al. (2023b)? Is it just the number of physical attacks and object detectors included in the evaluation? For physical attacks against traffic signs, what makes the author's work valuable over Hingun et al. (2023) work? Hingun et al. (2023) propose to model real-world conditions to better project and apply the patch in the image. Which of your or their benchmark is better suited to compare physical attacks against traffic sign recognition? Which of these benchmarks most accurately represents real-world conditions?

• A lack of details about how the datasets are generated, how the different physical attacks are projected into the scene. Did you use the physical attacks available in the GitHub that may be associated with the attack, or did you re-implement and design the attack yourself? On which object detectors are the different physical attacks optimized? How are the different ground truths boxes generated?

• The main weakness for me is the following. The authors express the need to model better cross-domain transformations $T_{P2D}(T_{D2P} (δ))$. I agree that this is an interesting and important research direction that may benefits the physical attack community. However, this paper does not propose a contribution to advance in this direction. What are the discrepancies between the simulations you used and real-world scenarios, considering that the used simulations may not accurately represent real-world conditions? Can I expect the ranking of physical attacks to remain consistent in real-world environments? Evaluations using the COCO dataset or simulations rely on digital experiments, and it is still unclear which serves as the best proxy for real-world conditions. To close the gap between numerical experiment and physical experiment, Hingun et al. (2023) propose to model the brightness of real-world scenes to better project the patch in the image. It would be a valuable contribution if the authors could provide such an experiment.

1. Some figures are hard to read, e.g. Fig 1. Some of the text is too small and blurred.
2. Some settings seem problematic. Please see the questions.
3. The paper lacks conclusions and inspirations drawn from benchmarking multiple attacks. For example, which technique is essential in improving adversarial effectiveness?