Enhancing Diffusion-based Unrestricted Adversarial Attacks via Adversary Preferences Alignment

We sincerely thank the reviewer for their constructive feedback. Below, we provide point-by-point responses to each of the comments.

---

Weakness 1 & Question 4:
1. Heavy computation cost on per-image LoRA fine-tuning.

2. It is not clear whether the fine-tuned LoRA on one image can work on other images.

3. I am not sure weather the good performance is due to something like over-fitting, and the model is highly optimized for one specific image.

Thank you for the reviewer’s detailed questions. We would like to emphasize that our ultimate goal is to generate adversarial examples with strong transferability and high visual consistency. The use of LoRA serves merely as a means to achieve visual consistency alignment. To address your concerns comprehensively, we provide clarifications from the following four perspectives:

1). Regarding the computational cost of per-image LoRA fine-tuning
a. Visual consistency alignment (VCA) is relatively lightweight, we update only a very small number of LoRA parameters (<1% of full parameters ), while keeping the backbone model frozen.

b. On an A100 GPU, fine-tuning for a single image takes approximately 38 seconds. In comparison, the null-text optimization used in the previous state-of-the-art ACA method takes around 55 seconds.

c. VCA only requires around 6GB of GPU memory, making it a lightweight module with minimal resource demands.

Overall, we believe that the computational cost is acceptable given the significant performance VCA gains (please see R#a1PE Question 1).

2). Regarding whether the fine-tuned LoRA can transfer to other images
Our method is fundamentally an image-specific adversarial attack framework, which optimizes a fine-tuning module (LoRA) for a given input image to preserve its high-level semantics. As such, we do not aim for generalization across entirely different images.

Under this setting, LoRA is inherently tailored to each image and is not intended to be transferable to arbitrary inputs — which aligns with our design objective.

Nevertheless, we believe that good transferability should exist among semantically similar images. To validate this, we conducted the following experiment:

• We randomly selected 5 video clips from UCF101 video dataset.
• From each clip, we extracted 16 consecutive frames.
• We trained a LoRA on the first frame of each clip using our VCA.
• Then we applied the same LoRA to subsequent frames to evaluate its reconstruction ability on semantically similar images.

The results are shown below:

We observe that the LoRA fine-tuned on the first frame generalizes well to the subsequent frames, suggesting that our VCA module exhibits good generalization across images with similar semantics. This observation indicates a potential future optimization: grouping batch data by semantic similarity and performing VCA once per group, rather than per image.

3) Regarding concerns about “overfitting”

We believe this concern stems from a misunderstanding: that overfitting the LoRA parameters to a single image would compromise the generalization of the attack.

In fact, our method fine-tunes the LoRA module in an image-specific manner to preserve semantic consistency with the input. The generalization, however, arises from the adversarial image itself, not from the LoRA parameters.

Specifically, we first align visual semantics via LoRA fine-tuning, then apply an alignment loss targeting the victim model to improve transferability. This two-stage strategy ensures both semantic preservation and strong cross-model attack success (see Table 1). Thus, while the optimization is per-image, the resulting adversarial image generalizes well across models.

4) On the advantages and justification of image-specific optimization

Image-specific optimization is a widely adopted strategy in both adversarial attacks and personalized generation. For instance, many diffusion-based attack methods and personalized generation[1] fine-tune models per image or subject to ensure high fidelity.

Compared to dataset-level optimization, image-specific tuning offers key advantages:

• Flexibility: A single alignment can generalize across multiple downstream tasks (see Appendix G.2).
• Higher performance ceiling: Tailoring to each image enables better use of its semantic features.
• Faster convergence: More stable and efficient without large-scale data.
• Low adaptation cost: Less dependent on task/model configurations than dataset-level methods.

[1]DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation, CVPR, 2023.

---

Weakness 2 & Question 3:
Limited discussion on why the proposed method can achieve closer Pareto optimality.

Thank you for your insightful comment. In multi-objective optimization, the Pareto optimal set consists of solutions where no objective can be improved without sacrificing another. Since the true Pareto frontier is intractable in our task, we construct the analysis shown in Figure 5(b) to approximate it. In this figure, the x-axis measures visual consistency, and the y-axis represents attack effectiveness. Ideally, Pareto-optimal solutions lie toward the top-right corner, where both objectives are maximized.

Our proposed algorithm, APA-SG (red points), first approaches the optimal solution space for visual consistency and then searches locally for solutions with higher attack effectiveness. Compared to other joint optimization methods (blue points), our approach yields results that are closer to the Pareto-optimal frontier (i.e., the upper-right region of the plot).

We will provide a more detailed discussion of this insight in the final version, specifically in the analysis of Section 5.5.

---

Weakness 3:
Some core operations are from references papers. For example, equation (7) is following ACA, and skip gradient also mentioned in ACA, and gradient checkpointing is widely used in network tuning. Although these are correctly cited, it would raise the concern of novelty.

Thank you for your comments. We would like to clarify that we never claimed these two gradient computation methods as our own contributions. These are standard components in attack optimization, much like how backpropagation is standard in model training. Our intention in describing these methods is to highlight the flexibility of our framework, which can readily incorporate different gradient computation strategies as needed.

While previous works have adopted these optimization techniques, none has systematically examined how different gradient computation strategies affect the attack performance — a gap our work addresses.

Most importantly, our main innovations are as follows:

(1) For the first time, we interpret the unrestricted adversarial example generation problem from the perspective of preference alignment.

(2) We decouple the conflicting preferences and propose a two-stage alignment framework that is highly flexible and adaptable to different tasks, diffusion models, optimization parameters.

(3) To address the unique needs of attack preference (i.e., strong attack generalization), we introduce dual-path attack optimization and diffusion augmentation, which improve attack preference alignment and prevent reward hacking.

In summary, the core novelty of our work lies in the new perspective, the general and flexible optimization framework, and the newly designed components for attack preference alignment—not in the specific gradient calculation methods themselves.

---

Question 1:
Correction of Eq. 6.

Thank you for your insightful comment. You are absolutely right — since a similar form was already introduced in Equation 2, we applied a simplified notation in Equation 6. As per your suggestion, we will revise the equation accordingly. We sincerely appreciate your feedback.

---

Question 2:
In Equation (7), why do we use the L1 norm

Thank you for your careful observation. Both ACA and our method follow the classical work in adversarial attacks — MI [1], which first introduced momentum into adversarial example generation. As stated in the original paper (last paragraph of Section 3.1), any distance metric can be used for normalization, with L1 norm being the default. Therefore, using the L2 norm is also valid.

[1] Dong et al., Boosting Adversarial Attacks with Momentum, CVPR 2018.

---

Question 5:
The manuscript had discussed the one-stage and two-stage alignment in page 9 line 320 to 326. It is not clear the performance of the alignment without LoRA. Please consider providing more details.

Thank you for your valuable comment. Following your suggestion, we have now included the corresponding quantitative results, shown in the table below, ResNet50 as the source model:

These results show that aligning only with attack effectiveness significantly boosts attack performance, but severely degrades image quality (CLIP Score = 0.62), making the outputs practically unusable. We will add these results to Figure 5(b) and discuss them on page 9, lines 320–326.

Once again, thank you for your time. If these clarifications resolve your concerns, we would be grateful if you could consider raising your score.

We sincerely thank the reviewer for their constructive feedback and for recognizing the effectiveness of our method and the thoroughness of our experiments. Below, we provide point-by-point responses to each comment.

---

Weakness 1 & Question 1.2:
The paper’s organization is challenging to follow, with an abundance of complex design choices that obscure the core methodology and hinder readability. Consequently, the paper's organization makes it difficult to grasp the necessity of the numerous proposed techniques. The authors are adviced to significantly improve the presentation clarity.

We thank the reviewer for the feedback. We will revise the manuscript to emphasize how each serves a distinct role in the alignment process, and de-emphasize optional parts to reduce perceived complexity, primarily in the following aspects:

1. Clarifying the core methodology
   We will restructure Section 4 to clearly present our two-stage alignment framework at the outset and add a concise overview summarizing the core pipeline.

• Stage 1 aligns visual consistency by encoding the input image into LoRA parameters.
• Stage 2 enhances attack transferability through dual-path optimization and diffusion augmentation.

2. Reducing perceived complexity
   While our framework includes several components, we will clearly distinguish between core modules (e.g., alignment stages, dual-path attack optimization) and optional extensions (e.g., different optimization parameters, gradient strategies). To improve readability, we will include a brief summary of these in the paragraph starting at line 230.

3. Highlighting simplicity and extensibility
   Our framework is easy to implement with minimal modifications to Hugging Face’s Diffusers library. It is decoupled from specific models and tasks, making it generalizable to a wide range of downstream applications.

We believe these revisions will significantly improve the paper’s clarity and accessibility.

---

Weakness 2:
The framework’s high time overhead significantly limits its real-world applicability.

We thank the reviewer for the concern regarding time cost, which we have acknowledged as a limitation in our paper. However, we would like to clarify two key points. First, our method offers a relative efficiency advantage over existing approaches. Second, in current applications of unrestricted adversarial examples, the primary focus lies in attack performance and image quality, with efficiency being a secondary yet desirable attribute. We further elaborate on this issue from four perspectives:

1. Competitive Efficiency Compared to SOTA

As reported in Appendix B, our method achieves approximately 16% speedup over the SOTA method ACA on a single A100 GPU, while significantly outperforming ACA in terms of attack success rate and visual quality (see Tables 1,2,3). This demonstrates a favorable trade-off between efficiency and performance.

2. Real-World Applicability Is Primarily Offline

Unrestricted adversarial examples serve two key real-world purposes: (1) as a tool for adversarial robustness evaluation, particularly under transfer-based attacks; and (2) as hard samples for data augmentation（recent studies[1,2] have shown that hard samples generated by diffusion models can substantially improve downstream model performance when used for training). Importantly, both applications are conducted offline and thus impose no strict real-time constraints on the generation process.

In summary, the practical value of unrestricted adversarial examples lies in their ability to stress-test and enhance models. Their generation time is largely irrelevant in these contexts, as performance—not speed—is the primary concern.

[1]ASAM: Boosting Segment Anything Model with Adversarial Tuning, CVPR,2024

[2]Synthesizing Near-Boundary OOD Samples for Out-of-Distribution Detection, ICCV,2025

3. Acceleration Strategies

The main overhead stems from attack effectiveness alignment, particularly the attack iterations and denoising steps during diffusion sampling. Thanks to the strong performance of our method, we can further reduce the number of attack iterations to improve efficiency (see Question 2 for details). Moreover, more efficient sampling strategies may offer further acceleration opportunities.

We have implemented a parallel version of our pipeline to support batch-wise optimization for attack alignment, substantially improving the efficiency of generating batched adversarial examples. The updated code will be released upon paper acceptance.

---

Question 1.1:
While the approach presents an interesting exploration of unrestricted attacks, the connection to adversary preference alignment appears tenuous.

Thank you for your comments. We would like to clarify that the key distinction between our APA and prior unrestricted attacks lies in our alignment-driven modeling. Specifically, our framework is motivated by the need to:

• define malicious preferences (visual consistency and attack effectiveness),
• quantify these preferences via rule-based or learned reward functions,
• select appropriate alignment strategies (DPO, reinforcement learning, or direct backpropagation),
• define stable alignment framework (joint or decouple)
• design task-specific modules such as dual-path optimization and diffusion augmentation.

These components are unified in our proposed two-stage alignment framework, where the central novelty lies in decoupling the alignment of inherently conflicting preferences—i.e., maximizing attack performance within the solution space constrained by visual consistency, thereby approaching closer to the Pareto frontier. Furthermore, dual-path optimization and diffusion augmentation are specifically introduced to mitigate overfitting issues commonly observed in direct backpropagation. We validate the effectiveness, flexibility, and robustness of our method through extensive experiments.

---

Question 1.3:
Solving an inverse problem (maximizing adv. loss) via denoising path guidance, similar to DPS.

Thank you for your suggestion. We carefully compared DPS with our method and found they address fundamentally different goals. DPS solves an inverse problem focused on reconstruction quality, whereas our attack alignment targets the generation of transferable adversarial examples, where reconstruction quality is already ensured by visual consistency alignment.

The only loose connection lies in using gradient-based guidance: DPS leverages reconstruction loss gradients, while our dual-path guidance is tailored for adversarial alignment. Notably, such guidance originates from classifier-guided diffusion, which we already compare against—along with its stronger variant, Upainting—both being conceptually closer to our method than DPS.

As shown in Figure 5(b), our dual-path guidance, incorporating a cleaner latent reference ($z_{in}$ in Eq. 10) and momentum accumulation, yields significantly better attack performance. We will include a discussion on DPS in the final version.

---

Question 2:
Although APA-GC demonstrates the best performance, the authors provide no analysis of its computational efficiency. It is crucial to compare the performance of different attack algorithms under equivalent computational budgets. The authors can include results using standardized metrics, such as ASR plotted against computation time (e.g., wall-clock time or FLOPs), with time/compute on the x-axis and ASR on the y-axis.

Thank you for your suggestion. First, we have already analyzed the time efficiency of our method compared to the state-of-the-art ACA in Appendix B. Under the same number of attack iterations and denoising steps, our method achieves higher efficiency and significantly better performance than ACA.

We fully agree with your point. As discussed earlier, the primary time cost of alignment-based attacks stems from the attack optimization and the diffusion denoising process. Since the number of attack iterations and denoising steps directly determines the number of forward/backward passes through the UNet, they serve as reliable proxies for overall compute cost. We therefore analyze both components separately to explore whether efficiency can be further improved without compromising performance.

1. Attack Iteration

Although we are unable to include images and figures, we report the transfer success rate of our method across different numbers of attack iterations (see the table below). For reference, ACA achieves 59.49 Avg.ASR.

Our APA-SG surpasses this benchmark within just 6 iterations, and APA-GC within only 4. This demonstrates that, thanks to the superiority of our approach, we can significantly reduce the number of iterations while still achieving state-of-the-art attack performance. In the final version, we will convert the above table into a curve plot.

2. Denoising Process

As shown in Figure 6(a) and 6(c) of the main paper, the number of denoising steps can also be further reduced. For example, using only 5 denoising steps (Figure 6a), our method achieves a 1.7× speedup while still reaching a success rate of 87.14, which remains substantially higher than ACA.

Once again, thank you for your time. We sincerely hope our response can address your concerns.

We sincerely thank the reviewer for their constructive feedback and for recognizing the novelty of our method and the thoroughness of our experiments. Below, we provide point-by-point responses to each comment.

---

Weakness 1:
Although the 2-stage method is novel, multi-component optimization process may introduce complexity in real-world implementation. Possibly further simplification or integration techniques could be beneficial.

Thank you for your valuable comment. We understand the concern regarding potential complexity in real-world implementation due to the multi-component design. We would like to clarify that our method is straightforward to implement and highly modular, as outlined below:

Simple and Modular Implementation: Unlike prior approaches that tightly couple objectives into a single optimization loop, our method is implemented as a clean two-stage pipeline. The visual consistency alignment is based on lightweight LoRA fine-tuning, and after merging the LoRA parameters into the UNet, no additional parameters are introduced for the subsequent attack alignment. The attack alignment is seamlessly integrated into the denoising process via plug-and-play modules (i.e., dual-path attack optimization + diffusion augmentation). The entire pipeline is fully compatible with the widely-used Diffusers library, requiring only a few lines of code to integrate with existing workflows.

Low Integration Cost: Despite its modularity, the default configuration works well across various settings, requiring minimal hyperparameter tuning. For most use cases, users only need to specify the gradient computation method (e.g., skip-step or gradient checkpointing), with the rest handled by default wrappers. The full method runs efficiently on a single RTX 3090 GPU without special hardware or distributed training.

We will revise the paper accordingly to make the simplicity of our implementation more transparent. Specifically, we will add a structural overview and an integration guide in the Appendix.

---

Weakness 2:
provide a summary table of symbols

Thank you for your comment. We have provided a detailed symbol table to aid your understanding.

Symbol Table

A more detailed description is provided in Appendix D, which can be best understood together with the accompanying pseudocode.

---

Weakness 3 & Question 3:
In Eq. 6, the switch from image similarity to latent similarity is abrupt; justification for why latent-space similarity is a good surrogate for visual consistency is not well articulated.

Regarding the shift from pixel-space similarity to latent-space similarity, we appreciate your observation. Our design follows the motivation outlined in the LDM paper[1], particularly the concept of perceptual compression—that is, most pixel-level information is redundant, and perceptually equivalent representations can be obtained in the latent space through a trained autoencoder (see Sec. 3.2 and Fig. 2 in the original paper). Therefore, we perform consistency optimization in the latent space, which preserves high-level semantics while improving training efficiency and stability. Although this substitution is not strictly equivalent, it is a common practice in the current image generation field.

To further assess the perceptual loss incurred by using latent representations, we conducted the following experiment:

VAE: Reconstruction quality of the input image using only the VAE encoder-decoder.

VAE + DDIM: Standard reconstruction quality using the full diffusion model pipeline.

VAE + DDIM + VCA: Reconstruction quality after incorporating our Visual Consistency Alignment (VCA) module.

We found that the VAE itself introduces only a negligible perceptual degradation (with a 0.03 drop in CLIP score and a 0.17% increase in Avg ASR). Furthermore, we observed that incorporating our VCA effectively compensates for the perceptual loss caused by denoising, reaching a level comparable to that of VAE compression.

Moreover, another advantage of compressing into the latent space is that optimizing the latent-to-pixel mapping does not introduce noticeable noisy perturbations, whereas direct optimization in the pixel space often does (as seen with DiffPGD in Figure 4).

[1] Rombach R. et al., "High-resolution image synthesis with latent diffusion models," CVPR, 2022.

---

Weakness 4
Details on LoRa Fine-Tuning, to be more precise, how LoRa modules are inserted.

Thank you for your rigorous review. As stated in Appendix F (lines 597–599), during the visual consistency alignment phase, we fine-tune only the projection matrices Q, K, and V in the UNet’s attention modules. We set the LoRA rank to 8 and train for 200 steps using the AdamW optimizer with a learning rate of 1×10⁻⁴. Before performing attack effectiveness alignment, we simply merge the LoRA parameters into the UNet, introducing no additional parameter overhead. For more implementation details, please refer to the source code included in the supplementary zip file.

---

Question 1
In the paper, you conducted evaluation on existing defenses which originally focusing on Lp attacks, but are you also considering defenses method on unrestricted adversarial attacks?

Thank you for your careful feedback. The defense methods Res-De (a model trained to reduce shape and texture bias) and Shape-Res (a model trained to increase shape bias for improved robustness) in Table 2 are specifically designed for unrestricted adversarial examples, as introduced in line 273. Compared to prior work, our defense evaluation is more comprehensive, covering adversarial training (for both CNNs and ViTs), input preprocessing defenses, diffusion-based defenses, and defenses tailored to unrestricted attacks. However, as shown in Table 2, existing methods still struggle to defend against our strong adversarial examples, which highlights a clear gap and motivates future research in this direction.

---

Question 2
In Eq.7, it introduces momentum-based gradients, but $\Pi_{z^0_T+\epsilon_a}$ , what this refer to? Can you provide more details?

In the adversarial attack domain, $\Pi_{z^0_T+\epsilon_a}$ is commonly defined as restricting the modification range of $z^0_T$ within $[ z^0_T-\epsilon_a, z^0_T+\epsilon_a ]$. Its purpose is to prevent the optimized $z_T$ from deviating too far from the original $z_T^0$. We will restate this more explicitly in the final version.

---

Question 4
The “skip gradient” and “gradient checkpointing” strategies are described briefly; their impact on optimization stability and convergence, I think is not well justified or analyzed. Are there convergence guarantees ?

Thank you for your rigorous review. To ensure a fair comparison with prior work, we fixed the attack iteration number $N$ to 10 steps and therefore did not include an explicit convergence analysis initially. Theoretically, gradient checkpointing yields more accurate gradient estimates than skip-gradient methods, which is supported by our experimental results showing that APA-GC outperforms APA-SG in attack effectiveness.

We supplement a convergence analysis as follows:
First, we use the white-box ASR as an indicator of convergence. In adversarial attacks, accurate gradients should result in a steadily increasing white-box ASR, reflecting effective optimization.
Second, we randomly sampled 100 images and recorded the reward values over 15 iterations to observe convergence behavior. The results are summarized in the table below:

We observe a rapid increase in reward from steps 2 to 8, followed by slower growth from steps 10 to 14. Together with the high white-box ASR at step 10, this suggests the optimization essentially converges within 10 steps. Moreover, APA-GC shows a clearly faster reward increase than APA-SG. Due to certain methodological constraints, we cannot provide a clearer convergence curve at this stage, but we will include it in the final version of the paper.

Once again, thank you for your time. We sincerely hope our response can address your concerns.

We sincerely thank the reviewer for their constructive feedback and for recognizing the technical contributions of our work. Below, we provide point-by-point responses to each comment.

---

Weakness 1:
1. APA still follows per-sample gradient-based optimization and does not perform true model-level alignment.

2. finding better solutions that simultaneously achieve high image quality and strong attack performance has long been a key objective in adversarial attacks.

3. The so-called attack reward are essentially standard loss functions like cross-entropy.

Thank you for your comments. We would like to clarify that the key difference between APA and conventional attack optimization lies in its alignment-driven design. Our framework is motivated by how to:

• define malicious preferences (visual consistency & attack effectiveness),
• quantify them (rule-based or learned rewards),
• select alignment strategies (DPO, RL, or direct backpropagation),
• design task-specific modules (dual-path optimization & diffusion augmentation).

These considerations lead to our proposed two-stage alignment framework. Below, we address the three concerns you raised in detail:

1. Instance-level alignment:
   You're right, we perform per-instance alignment, not model-level alignment, as a practical choice due to the difficulty of generating unrestricted adversarial examples under conflicting preferences. This approach aligns with test-time preference alignment [1].

   [1] Test-Time Preference Optimization: On-the-Fly Alignment via Iterative Textual Feedback, ICML,2025.

2. No novelty claim on objective formulation:
   We do not claim that our objective design is novel. Instead, we extract preferences from adversarial attack literature. Our key contribution is to decouple the optimization of these preferences, showing clear advantages over joint optimization.

3. Reward function is not our contribution:
   Under the direct backpropagation setting, it is indeed mathematically valid to formalize a reward function as a loss. Actually, reward design is not our contribution (discussed in Section 4.2). Our key contributions under this setting lie in dual-path attack optimization and diffusion augmentation, which are specifically designed to address known issues with direct backpropagation (such as overfitting to the reward model).

In summary, we believe the technical similarity between alignment via backpropagation and gradient-based optimization in adversarial attacks may have led to some misunderstanding regarding the novelty of our work. However, the core innovation of APA is the decoupled alignment of two preferences (instead of joint optimization) and the design of task-specific modules that effectively mitigate overfitting issues in direct backpropagation settings. We further support our claims through extensive experiments demonstrating the superiority, flexibility, and robustness of our method.

---

Question 1:
Except for the results in Figure 3, are there any quantitative results or theoretical analyses to support the effectiveness of Visual Consistency Alignment?

We sincerely appreciate the reviewer’s careful analysis. To further clarify the role of our proposed Visual Consistency Alignment (VCA), we have conducted two additional ablation studies.

1. To evaluate whether VCA improves the inherent reconstruction ability of diffusion models, we compare DDIM with and without VCA under a non-attacking setting.

   Method	LPIPS↓	SSIM↑	CLIP Score↑	NIMA AVA↑	CNNIQA↑	Avg ASR↓
   DDIM	0.19	0.73	0.89	5.03	0.63	22.10
   DDIM+VCA	0.05	0.85	0.97	5.13	0.63	7.60

These results suggest that VCA clearly enhances the reconstruct quality of diffusion model.

2. We also evaluated the quality of images generated when only the attack effectiveness alignment (i.e., without VCA):

   Method	LPIPS↓	SSIM↑	CLIP Score↑
   APA-SG w/o VCA	0.55	0.46	0.62
   APA-SG	0.25	0.67	0.86

The noticeable degradation in visual quality upon the removal of VCA highlights its essential contribution to maintaining visual consistency during attack alignment. We will incorporate all the above findings into the revised paper to better communicate the role and advantages of VCA.

---

Question 2.1:
APA-SG’s images in Figure 4 look better than ACA and DiffPGD-MI despite similar metrics, yet Appendix H shows artifacts. Were the visuals selectively chosen? Please provide more random samples.

Thank you for your careful observation. First, we would like to clarify that we did not deliberately select qualitative examples. Due to the constraints, we could not include the full set of test results. However, you can reproduce them using the provided code in the supplementary materials. We will release all results, including those of the baselines, upon acceptance of the paper.

In addition, we believe that our quantitative analysis is consistent with the qualitative findings. For instance, DiffPGD-MI introduces noticeable background perturbations while preserving image structure, leading to high SSIM but low aesthetics. ACA's low CLIP score reflects its tendency to alter image semantics. In contrast, APA-SG mainly modifies the background while preserving the subject, producing more natural outputs and thus a higher aesthetic score.

---

Question2.2 and Question3:
1. Could the authors clarify why the detailed results of APA-GC-P, which appear to offer better image quality but lower ASR, are not included in Tables 1 and 2?

2. Comparison with the state-of-the-art LDM-based attack DiffAttack [1] seems to be missing."

Thank you for your valuable suggestions. We have added detailed results for APA-GC-P and DiffAttack.
For (a) and (b), we report the Avg. ASR due to space limitations. The complete results will be included in the final version.

a. Attack Performance Comparison (Table 1)

b. Attacks on Adversarial Defense (Table 2)

c. Visual Quality Comparison (Table 3)

Conclusion:

1. Our APA-SG and APA-GC achieve 10.09 and 22.04 higher attack performance than DiffAttack, respectively.

2. Under defense scenarios, APA-GC, APA-SG, and APA-GC-P all demonstrate better robustness compared to DiffAttack.

3. APA-GC achieves comparable visual similarity and aesthetic scores to DiffAttack. Moreover, APA-GC-P exhibits significantly better visual quality than DiffAttack with comparable attack performance. We will include qualitative examples in the main paper as well (we observed that images generated by DiffAttack often have overly sharp object boundaries).

4. These results further demonstrate the superiority of our two-stage design in APA over the joint optimization used in DiffAttack, enabling more effective and controllable generation.

5. As noted, APA-GC-P offers superior visual quality but moderate attack effectiveness. We include it to showcase the flexibility of our framework in supporting task-specific optimization choices.

---

Question4:
It should be clarified whether the APA variant compared with AdvDiff in Appendix E.3 remains under an untargeted attack setting, to avoid potential unfair comparisons, or how APA can be extended to targeted attack scenarios.

Thank you for your insightful comments. As stated in its original paper('Evaluation metrics' on page 10), both AdvDiff and AdvDiff-untarget (which enhances transferability) are intended for evaluating untargeted attacks. Accordingly, all comparisons were performed under the untargeted setting, with fairness across all baselines strictly maintained. Moreover, extending APA to targeted attacks is straightforward—one only needs to replace the optimization objective accordingly. Please refer to lines 612–619 for implementation details.

---

Question5:

APA appears more complex and involves more computation steps than ACA, yet runs faster according to Appendix B

Thank you for your suggestion. We identified two contributing factors. First, our VCA optimization is significantly faster than APA’s null text optimization (e.g., 38 s vs. 55 s). Second, our Stage-2 alignment is implemented on a lightweight codebase based on Diffusers, whereas ACA relies on a heavier framework, which introduces slight overhead. You may verify this using the code provided in the supplementary zip. Our codebase is clean and modular—easily integrable into Diffusers, with flexible parameter selection and customizable optimization strategies.

Regarding the complexity analysis: Stage 1 has a time complexity of O(n) (number of training steps), while Stage 2 is O(N·T) (attack iterations × denoising steps).

Once again, thank you for your time. We sincerely hope our response can address your concerns.