AngleRoCL: Angle-Robust Concept Learning for Physically View-Invariant Adversarial Patches

1. Attack results on other target objects

Response:

Thank you for the insightful question. To address the concern, we conducted additional experiments on two other target objects to demonstrate the broader applicability of our method.

Experimental Design: Following NDDA's evaluation scope, we selected fire-hydrant and horse as additional target objects to evaluate object-specific training capabilities. These objects represent different categories—urban infrastructure (fire-hydrant) and natural subjects (horse)—allowing us to assess our method's versatility across diverse visual domains.

Object-Specific Training Experiment: We trained dedicated angle-robust concepts for the two target objects, respectively, using their respective NDDA prompt templates. Then, we embed the learned angle-robust concepts into the NDDA prompt to generate angle-robust adversarial patches. Table R-3 presents the comparative AASR results with/without AngleRoCL:

Table R-3: Object-Specific Angle-Robust Concept Evaluation

Note: All experiments in Table R-3 were conducted in Digital Environment 1. For each NDDA prompt, we generated 5 patches, and the reported results represent the average performance across all generated patches.

The results demonstrate that our framework successfully generates effective angle-robust concepts across different target objects, achieving consistent improvements in all categories. This validates that the angle robustness challenge is not unique to stop signs but represents a broader limitation in existing T2I adversarial patch generation methods.

2. Pseudo-code and algorithmic details

Response:

Thank you for this valuable suggestion. We present the complete pseudo-code for our AngleRoCL training process as follows:

The training process follows a standard textual inversion framework but incorporates angle-aware supervision. For each training iteration, we generate adversarial patches using the learned concept embedding, apply perspective transformations to simulate multi-angle observations (lines 1-20), and optimize the embedding based on detection feedback across all angles (lines 14-16). The key advantage is that only the concept embedding is updated while preserving all original model parameters, ensuring stability and enabling the learned concept to generalize across different prompts and environments.

Algorithm: Angle-Robust Concept Learning (AngleRoCL)

Input:
  - Pre-trained Stable Diffusion model M (text encoder, UNet, VAE)
  - NDDA prompt templates P = {p_1, p_2, ..., p_N}
  - Training angles Θ_train = {θ_1, θ_2, ..., θ_V}
  - Placeholder token <angle-robust>, initializer token "robust"
  - YOLO detector D, perspective transformer T
  - Detection threshold τ, scaling factor λ
  - Repeat factor R for data augmentation

Initialization:
1. Add <angle-robust> token to tokenizer vocabulary
2. Initialize embedding E_<angle-robust> ← E_robust (copy initializer embedding)
3. Freeze all model parameters except E_<angle-robust>
4. Create training dataset: D_train = {(p_i, r) | p_i ∈ P, r ∈ [1,R]}

Training Loop:
for epoch = 1 to max_epochs do:
    for each batch in DataLoader(D_train) do:
        // Text encoding with learned concept
        1. angle-robust_embed ← TextEncoder(<angle-robust>)
        
        // Diffusion-based patch generation
        2. Initialize random latents z_0 ~ N(0,I)
        3. z_0 ← z_0 * scheduler.init_noise_sigma
        4. for timestep t in scheduler.timesteps do:
        5.     z_input ← concat([z_0, z_0])  // For classifier-free guidance
        6.     z_input ← scheduler.scale_model_input(z_input, t)
        7.    ε_pred ← UNet(z_input, t, angle-robust_embed)
        8.    ε_uncond, ε_text ← chunk(ε_pred, 2)
        9.    ε ← ε_uncond + γ * (ε_text - ε_uncond)  // CFG with scale γ
        10.    z_0 ← scheduler.step(ε, t, z_0).prev_sample
        11. end for
        12. P ← VAE.decode(z_0 / 0.18215)  // Generated adversarial patch
        
        // Batch multi-angle transformation
        13. transformed_views ← T(P, Θ_train)  // Transform to all V angles simultaneously
        
        // Detection and loss computation
        14. detection_output ← D(transformed_views)
        15. confidences ← ExtractMaxProb(detection_output)
        16. loss ← mean(max(τ - confidences, 0)) * λ
        
        // Backpropagation
        17. loss.backward()
        18. optimizer.step()  // Update only E_<angle-robust>
        19. optimizer.zero_grad()
        
        // Preserve original embeddings
        20. E_other ← freeze(E_original[other_tokens])
    end for
end for

Output: Learned embedding E_<angle-robust>

3. Discussion on potential defense solutions

Response:

Thank you for this valuable suggestion. Indeed, it is meaningful to discuss potential defense solutions in the broader impact statement. Based on previous works and our preliminary studies, we identify the following defense approaches: (1) Adversarial training methods that fine-tune detectors using our generated adversarial patches. However, this solution may reduce detection accuracy on other object categories and introduce new vulnerabilities. (2) Data purification methods that pre-process input images to disable adversarial patches by removing adversarial patterns. While effective for noise-like adversarial examples, our preliminary results show limited effectiveness against our patches. (3) Adversarial patch detection models that identify and crop patches to avoid their influence. However, this approach requires the detection model to be computationally efficient and demonstrate high generalization across unknown adversarial patches, which necessitates further careful study. We will incorporate and expand upon these defense discussions in our broader impact statement to provide a more comprehensive treatment of both the risks and potential mitigation strategies associated with our work.

4. Writing quality and formatting issues

Concern: Various writing issues, including missing closing bracket on Line 60, unclear sentences on Lines 67-69, short paragraphs on Lines 125-141, making text difficult to read, and confusing point descriptions.

Response:

We appreciate the reviewers' detailed feedback on specific issues, including missing brackets, unclear sentences, and fragmented paragraphs. We will thoroughly polish the manuscript to address these writing issues and ensure a consistent, clear presentation throughout the submission to avoid similar situations in the revised version.

For example, we will revise the unclear sentences (Lines 67-69) as:

Physical adversarial patches pose severe threats to critical systems, including autonomous driving [42], classification models [4, 12, 47], and other safety-critical domains [8, 9, 16, 36, 38]. These patches are highly reproducible and deployable [34], making it essential to investigate their attack effectiveness [4, 38], ....

5. More limitations discussion

Response: Thank you for the suggestions. In our original submission, we discussed the limitations on line 367 to 371. To further enhance this part, we have the following revision:

Limitations. Although our method significantly enhances the angle robustness of generated patches, the current version has several limitations that warrant further investigation. (1) Our training is constrained to only 9 sampled angles in the horizontal plane, which simplifies the optimization process but does not fully capture continuous 3D angle variations, including vertical perspectives commonly encountered in real-world scenarios. (2) The current version employs only YOLOv5 for the objective function, which may limit performance improvements on other detection architectures. Our preliminary exploration indicates that combining different detectors during concept training fails to achieve higher angle robustness, suggesting that a deeper study of multi-detector optimization strategies is required. (3) The learned concept presents low transferability across distinct object categories, indicating that developing methods with higher cross-category transferability represents an important direction for future research.

Important Clarifications on Factual Errors:

We appreciate your detailed review, but we must first address significant misunderstandings in your summary: (1) Distillation Model Misconception: We employ textual inversion, not distillation models. (2) Training Angle Sampling: We use 9 fixed angles during training, not -90° to 90° with 1° step (that's our testing protocol in Section 5.1). The response to the main concerns is as follows:

1. Paper clarity and methodology description

Objective: We generate standalone adversarial patches using T2I models that fool object detectors across multiple viewing angles in digital & physical environments. These patches cause misclassification—making detectors incorrectly identify our generated patch as a legitimate target object (e.g., "stop sign"). The term `adversarial patch' refers to a standalone image/object that is generated to be detected as a target object, rather than a modification applied to an existing object.

Motivation: Existing T2I patches only work within narrow viewing ranges. Current methods like NDDA succeed at frontal views (0°) but fail at oblique angles (±60°+), limiting real-world applicability (See Fig. 1). Task-specific instructions fail to improve robustness.

Methodology: AngleRoCL learns an <angle-robust> token that, when incorporated into prompts like "a photo of a <angle-robust> stop sign," guides diffusion models to generate patches maintaining effectiveness across wide viewing angles.

We will revise the paper for broader accessibility.

2. Hyperparameter discussion

Our approach builds upon textual inversion with task-specific hyperparameters: sampled angles and <angle-robust> token initialization. The results in the following two Tables show that 9 sampled angles provide better performance, and "robust" significantly outperforms alternatives by 20+ percentage points due to the alignment with the desired property.

Table R-4: Ablation study on different numbers of training angles for concept learning against YOLOv5. The AASR is the evaluation metric.

**Table R-5: Ablation study on different initialization tokens for <angle-robust>. **

3. Physical experiment constraints

We'd like to clarify the distinction between digital and physical experimental constraints, as the primary bottleneck lies in data collectionrather than processing.

Scale Comparison:

• Digital: 1,268,505 automated evaluations (1,401 patches × 181 angles × 5 detectors)
• Physical: Manual printing, positioning, and photographing from multiple angles. Testing 40 patches requires 2 full days. Replicating a digital scale would need ~70 working days plus significant resources.

Validation Purpose: Physical experiments demonstrate real-world transferability under authentic conditions (printing artifacts, lighting variations), not statistical equivalence to digital experiments. Our 180 physical patches significantly exceed existing literature validation while remaining feasible.

4 Generalization to other target objects

To demonstrate broader applicability, we conducted experiments on two additional objects (fire-hydrant and horse). Table R-3 shows consistent improvements across different object categories. Please refer to the response to the 2nd response to Reviewer KqvV.

5. Projective transformation function details

We provide the mathematical description of our projective transformation function Proj(·) used in Eq. (3).

Mathematical Formulation:

The function Proj( ) transforms the frontal-view image $\mathbf{I}_{0^\circ}$

to simulate angle $\theta$ observations via homography matrix $\mathbf{H}_\theta$. We have

$\mathbf{I}_\theta (x, y) = \mathbf{I}_0^\circ \left( x', y' \right)$

where

where $M_{ij}$ are elements of $\mathbf{H}_\theta$.

$\mathbf{H}_\theta$ Calculation Steps:

1. Calculate camera position from angle $\theta$;
2. Compute view matrix $\mathbf{V}$ (world-to-camera coordinates);
3. Obtain projection matrix $\mathbf{P}$ (camera-to-2D coordinates);
4. Combine $\mathbf{V}$ and $\mathbf{P}$ for $\mathbf{H}_\theta$.

This framework maintains differentiability for gradient optimization during AngleRoCL training.

6. Patch selection in experiment and completed results

"Remove Text or Pattern" prompt definition: These prompts are sourced from the NDDA dataset and involve modifications that remove robust features from stop signs, as detailed in our supplementary materials.

Patch selection process: Following NDDA and MAGIC protocols, we generated 50 patches for each of the 8 prompts, totaling 400 patches. We randomly selected 100 patches from this pool for our main experiments.

Complete Dataset Validation: We conducted supplementary experiments using the complete 400-patch dataset to address the concern. Table R-6 demonstrates that the results via our randomly sampled patches accurately represents the complete dataset performance.

Table R-6: Complete Dataset Validation

7. Performance difference between Table 1 and Supp Mat Table 1

Performance differences between digital (Table 1) and physical environments stem from substantial environmental variations.

Environmental Factors: Digital uses the nuImage dataset scenes while physical experiments use different geographic locations (different countries), weather conditions, and camera specifications.

Cross-Validation: We digitally evaluated patches in physical environment images to verify consistency: In Table R-7, when physical environments are evaluated digitally, performance ranking (MAGIC+AngleRoCL > NDDA+AngleRoCL) matches physical results, confirming environmental factors drive differences.

Table R-7: Digital-Physical Environment Cross-Validation. AASR is as the metric

8. Angle span reduction in real-world settings

The angle span reduction from ±90° to ±70° was based on two key considerations: efficiency optimization and empirical observations from our digital experiments.

Justification: Figure 2 shows most methods have substantially reduced performance beyond ±75°, where geometric visibility constraints dominate over patch design differences. The ±70° range captures meaningful performance variations. Given the significant time and labor costs of physical experiments (as discussed in our previous response), we prioritized the angle ranges where method differences are most pronounced rather than testing angles where all methods uniformly fail.

Extended Validation: We tested the full ±90° range to address concerns. The results show that including extreme angles has minimal impact on conclusions, validating our original angle range selection.

Table R-8: Extended physical validation (-90° to 90°)

9. Justifications

(1). "25 matched pairs" means pairs with identical settings except for our <angle-robust> token presence/absence. (2). Normalized weight in Sec. 5.1 means uniform weighting: $w(\theta) = \frac{1}{|\Omega|}$ for all angles, meaning equal contribution to AASR calculation. (3). Figure 4 concludes that the <angle-robust> concept is associated with visual features. We will improve all figures for better readability.

1. Practical safety significance of multi-angle attacks

Response:

Thank you for this thoughtful concern about practical safety implications. While frontal views are indeed primary for driving decisions, angle robustness represents a significant real-world security challenge supported by both practical scenarios and extensive research attention.

Angle Robustness is Safety-Critical in the Real World:

• Traffic signs at intersections are viewed from various angles as vehicles approach from different directions. For example, in the highway merging and lane changes, drivers observe traffic signs at oblique angles when merging or changing lanes around construction zones and temporary traffic control.

• Autonomous vehicle sensor arrays: Modern AVs like Tesla, Waymo use 8+ cameras covering 360° views, continuously processing visual information from multiple perspectives simultaneously—angle-robust patches could compromise their entire perception system.

• Even "frontal" approaches involve angle variations due to road curvature, vehicle positioning, and sensor placement. In particular, for the emergency scenarios: First responders and emergency vehicles often approach intersections from non-standard angles where traditional patches would fail.

Extensive Research Validation: The critical importance of angle-robust adversarial attacks has been acknowledged by substantial research investment: multi-view adversarial examples [2, 12], 3D adversarial patches [7, 21], viewpoint-invariant adversarial noise [35]. This sustained research focus across premier venues (CVPR, ICCV, USENIX Security, NeurIPS) demonstrates that the research community recognizes angle robustness as a fundamental security challenge requiring urgent attention, rather than merely a theoretical concern.

2. Attack results on other objects

Response:

Thank you for the insightful question. To address the concern, we conducted additional experiments on two other target objects to demonstrate the broader applicability of our method.

Experimental Design: Following NDDA's evaluation scope, we selected fire-hydrant and horse as additional target objects to evaluate object-specific training capabilities. These objects represent different categories—urban infrastructure (fire-hydrant) and natural subjects (horse)—allowing us to assess our method's versatility across diverse visual domains.

Object-Specific Training Experiment: We trained dedicated angle-robust concepts for the two target objects, respectively, using their respective NDDA prompt templates. Then, we embed the learned angle-robust concepts into the NDDA prompt to generate angle-robust adversarial patches. Table R-3 presents the comparative results with/without AngleRoCL:

Table R-3: Object-Specific Angle-Robust Concept Evaluation

Note: All experiments in Table R-3 were conducted in Digital Environment 1. For each NDDA prompt, we generated 5 patches, and the reported results represent the average performance across all generated patches.

The results demonstrate that our framework successfully generates effective angle-robust concepts across different target objects, achieving consistent improvements in all categories. This validates that the angle robustness challenge is not unique to stop signs but represents a broader limitation in existing T2I adversarial patch generation methods.

3. "Direct Optimization" definition in ablation study

Response:

Thank you for your comments. "Direct optimization" refers to a baseline approach like [35] where we directly optimize adversarial patch pixels using backpropagation to maximize detection confidence across multiple viewing angles, rather than learning a reusable text embedding concept. To further address the concerns, here are the setup details of the baseline:

Training Setup:

• Optimization Target: Patch pixel values (3×512×512 tensor) vs. AngleRoCL's text embeddings
• Loss Function: Same loss function as AngleRoCL
• Angles: Same training angles as AngleRoCL
• Detector: Same YOLOv5 feedback as AngleRoCL
• Optimizer: Adam optimizer vs. AngleRoCL's AdamW on embeddings

4. Choice of textual inversion instead of LoRA

Response:

Thank you for this insightful question. We chose textual inversion over LoRA for several principled reasons related to our method's core objectives and practical considerations.

1. Plug-and-Play Capability: Textual Inversion enables seamless integration without affecting base model functionality - users simply load our trained embedding and modify prompts from "a blue square stop sign" to "a <angle-robust> blue square stop sign" to activate angle robustness. LoRA, despite using low-rank adaptation theory (decomposing weight updates as ΔW = BA where B∈R^(d×r), A∈R^(r×k), r<<d), modifies UNet layers and affects all generations even when angle robustness is not desired, compromising the model's normal text-to-image capabilities.

2. Concept Interpretability: Textual Inversion enables direct semantic analysis of learned robustness concepts through embedding analysis, as demonstrated in our discussion (Line 329 - 349) where we show the learned concept develops meaningful associations with color and shape features, providing insights into which visual attributes contribute to angle robustness. LoRA distributes learned features across multiple transformer layers, making it significantly more challenging to isolate and interpret the specific concept being learned.

3. Training Stability and Parameter Efficiency: Textual Inversion only trains the <angle-robust> embedding approximately 768 parameters while preserving all other embeddings and network parameters. LoRA trains more parameters (797,184 parameters, 0.09% of UNet with rank=4). However, we observed poor fitting performance that led to severe concept drift during our preliminary experiments - the learned representations progressively deviated from stop sign generation, resulting in images that could barely be detected as stop signs, severely compromising generation quality. We believe a novel learning strategy should be studied for the LoRA for angle-robust generation.

Our approach prioritizes learning transferable, interpretable concepts that enhance rather than modify the base model's capabilities, making textual inversion the better choice for our angle robustness objective.

Thank you for providing author rebuttal. The reviewer thinks the paper is enough to be accepted and solve interesting problem. The authors also provided additional experiments that the reviewer requested and it works well. However, despite of the rebuttal about practical significance of multi-angle attacks, the reviewer still thinks the problem is only useful for specific situation (e.g. intersection and road curvature). Therefore, the reviewer maintains the original rating.

1. YOLOv5 Training Bias, Model-Agnosticism Claims, and Multi-Detector Training

Response:

Thank you for this insightful question about detector-specific bias. We appreciate your observation and would like to clarify our claims and provide a detailed analysis.

Clarification of Model-Agnosticism Claims: Our original claim about the "model-agnostic" referred to the ability to achieve consistent angle robustness improvements across unseen, black-box detectors during testing, rather than uniform improvement magnitudes across all architectures. The terminology may be misleading, and we will revise our manuscript to provide a clearer characterization of our approach's capabilities and limitations. While we use YOLOv5 to guide concept learning during training, we evaluate on four additional detectors (Faster R-CNN, YOLOv3, RT-DETR, and YOLOv10) that provide no guidance information during the generation process. As demonstrated in Table 1 in the submission, AngleRoCL enhances performance across all tested detectors in six environments, demonstrating cross-detector transferability.

Regarding Multi-Detector Training: Intuitively, training with multiple detectors should enable learning a concept that achieves robust enhancement across different detector architectures. However, our implementation revealed that multi-detector training using both YOLOv5 and RT-DETR simultaneously encounters significant optimization challenges, with the loss function failing to converge effectively. As demonstrated in Table R-1, our single-detector training approach achieves substantially higher Angle-Aware Attack Success Rates (AASR) compared to the multi-detector variant, suggesting that naively combining multiple detector objectives creates conflicting optimization signals. These findings indicate that developing effective multi-detector training strategies represents an important avenue for future research, as the challenge likely stems from fundamental differences in how various detector architectures process visual features. While our current single-detector approach demonstrates strong cross-detector generalization, developing truly detector-agnostic angle-robust concepts remains an open challenge.

Table R-1: Multi-detector training vs single-detector training across detectors

Note: All experiments in Table R-1 were conducted in Digital Environment 1. For each NDDA prompt, we generated 5 patches using the trained robust concept, and the reported results represent the average performance across all generated patches.

2. Comparisons with Recent Diffusion-based Methods

Response:

Thank you for providing two related works. However, DiffPatch and Diffusion to Confusion (D2C) address fundamentally different research objectives compared to our method. DiffPatch and D2C focus on generating adversarial patches for human evasion (e.g., T-shirt patches to avoid person detection), while our work specifically targets angle robustness for object detection attacks—a previously unexplored dimension in T2I adversarial patch generation.

Our Comparison Strategy: While direct comparison is challenging due to divergent objectives, we evaluate these methods on angle robustness using our proposed AASR metric to demonstrate the critical gap in existing T2I adversarial patch methods.

Implementation and Initial Results: We contacted both research teams for code access. The DiffPatch authors graciously provided their implementation, enabling us to reproduce their results and evaluate the generated patches for angle robustness under our experimental protocol. Unfortunately, the D2C authors have not responded and no public code is available, limiting our comparison scope to DiffPatch.

Table R-2: Angle Robustness Comparison with DiffPatch

Note: The experiments in Table R-2 were conducted using patches trained on the INRIA person dataset with YOLOv5 as the target detector. We applied horizontal projective transformations same as our AngleRoCL to the patches and evaluated them against test set images as backgrounds to assess angle robustness performance.

Our results in Table R-2 demonstrate that DiffPatch exhibits significant angle robustness limitations when evaluated across multiple viewing angles—performing similarly to baseline methods like NDDA that lack angle-aware optimization. This comparison validates our core contribution: existing T2I adversarial patch methods, regardless of their sophistication in other dimensions, fail to address the critical challenge of maintaining attack effectiveness across diverse viewing angles in physical deployments.

Future Work Integration: The complementary nature of these approaches presents exciting opportunities for future research. Our angle-robust concept learning could be integrated with DiffPatch's customization framework to create patches that are simultaneously natural-looking, customizable, and angle-robust. Additionally, comprehensive evaluation including patch realism assessments using user preference studies and naturalness metrics would provide a more complete comparison framework. We will add the discussion into our revision and cite the two works.