Dual-Flow: Transferable Multi-Target, Instance-Agnostic Attacks via $\textit{In-the-wild}$ Cascading Flow Optimization

Thank you for your thoughtful review and detailed feedback on our paper. Below, we address the key points you raised:

The Dual-Flow framework may be complex for practical implementation, which could limit its accessibility for practitioners in the field. We appreciate you raising this important point. While the conceptual framework involves two flows, the practical implementation is designed to be modular and manageable. Our approach hinges on two main components:

1. A Pre-trained Diffusion Model: We leverage a publicly available, off-the-shelf diffusion model (Stable Diffusion), which practitioners do not need to train from scratch.

2. A Lightweight Adversarial Flow: The core of our training is fine-tuning a velocity function using Low-Rank Adaptation (LoRA). LoRA is specifically designed to be parameter-efficient, only requiring the training of a small number of additional parameters. This significantly reduces the complexity and resource requirements compared to training a full model.

Once our LoRA is trained, the inference process is very simple. We can directly use the image-to-image pipeline from existing code repositories (such as diffusers) to complete the process. It can be understood as performing DDIM Inversion on the original image, or the process of first adding noise then denoising, to generate adversarial samples.

To further improve accessibility, we will add a detailed implementation guide in the appendix of our revised manuscript, clarifying the steps and highlighting the framework's modularity.

The paper could benefit from a more detailed analysis of the computational costs associated with the proposed method compared to existing approaches.

Thank you very much for your suggestion on this! Here we tested the training time required for the baseline methods C-GSP and CGNC on multi-target attack tasks using the same RTX 3090 GPUs as ours. Specifically, we require one day of training on two RTX 3090 GPUs, corresponding to 48 GPU hours, while this number is 50 GPU hours for C-GSP and 250 GPU hours for CGNC. This demonstrates that our model achieves superior baseline performance while maintaining excellent training efficiency. We will include this content and more detailed analysis in the final version.

Focused solely on image classification. Extensions to other domains (e.g., object detection, NLP) are unexplored but noted as future work.

We thank the reviewer for these forward-looking comments and question. Indeed, our current work establishes the Dual-Flow framework's effectiveness within the domain of image classification, which is a standard and foundational benchmark for new adversarial attack methods.

We are very optimistic about the framework's potential for broader applications. The core principle of Dual-Flow—using a general pretrained flow for initial perturbation and a specialized, fine-tuned reverse flow for task-specific optimization—is highly adaptable.

• For object detection, the adversarial loss function in the reverse flow could be modified to suppress or alter bounding box predictions.

• For semantic segmentation, the goal could be to force specific pixels to be misclassified.

• For NLP, since it differs significantly from our domain, we will continue to explore in the future how to apply similar ideas to related tasks.

These are exciting avenues for future research that we intend to explore. We will emphasize these potential extensions more explicitly in the discussion and conclusion of the revised paper.

Are there any plans to extend the Dual-Flow framework to other domains or types of data beyond the current focus on image classification?

See the previous content.

Besides the attack success rate, could the authors consider including additional metrics (e.g., precision, recall) to provide a more comprehensive evaluation of the method's performance?

• We appreciate the suggestion to provide a comprehensive evaluation. For the specific task of targeted adversarial attacks, the Attack Success Rate (ASR) is the most direct and universally adopted metric in the literature. Additionally, we have also included 95% confidence intervals in the appendix for verification.

• The goal of a targeted attack is singular: for a given input image, to mislead a model into predicting one specific, pre-determined target class. Therefore, the outcome for each attack is binary: it either succeeds (the model outputs the target class) or it fails. The ASR, which is the percentage of successful attempts over all tests, precisely captures the efficacy of the method in achieving this goal.

• Metrics like precision and recall are standard for evaluating the performance of a classifier across a dataset with a ground-truth distribution of multiple classes. However, in our evaluation setting, we are not evaluating a classifier but the success of an attack. There is no concept of "true positives," "false positives," or "false negatives" in the traditional sense; there is only "attack success" or "attack failure" for the chosen target. Using ASR ensures our results are directly and fairly comparable to the established body of work on this topic. We will add a sentence in the experimental setup to clarify the rationale for using ASR as the primary metric in the context of targeted attacks.

• We will continue to explore adding more metrics in the future to more comprehensively evaluate model performance. Thank you for your suggestion!

We value your insights and believe they will significantly enhance our work's clarity and impact.

Thank you for your thoughtful review and detailed feedback on our paper. Below, we address the key points you raised:

1. Victim Models

• We completely follow the settings and models of baseline methods, maintaining complete consistency with them. For the victim models we used, all models employ the official weights provided for the 1000-class ImageNet dataset. For normally trained models, we directly call these models and their weights through the torchvision or timm libraries. For robust models (adversarially trained models): Inc-v3-adv and IR-v2-ens, we call their official weights through the timm library; for Res50-SIN, Res50-IN, and Res50-fine, we use the weights provided by the open-source project Texture-vs-Shape; for Res50-Aug, we use the official weights provided by the open-source repository AugMix.
• Since all the classifiers we use are pre-trained with official weights rather than training these models from scratch ourselves, the training details and accuracy information of these models are not reported by us. Thank you for your suggestion, we will supplement these details as much as possible in the final version!

2. Baseline Methods

• Thank you for your suggestion regarding the timeliness of baseline methods! We would like to point out that the CGNC method you mentioned was published at ECCV 2024. Unfortunately, for generative Instance-Agnostic Attacks methods, as of our paper submission deadline, we were unable to find more recent comparable open-source methods.

3. Content Reported in Tables

• Thank you for your careful observation! However, we must point out that those instance-specific attacks do appear in Table 1 but not in Table 2, but this is actually because for these methods, there is no distinction between multi-target attacks or single-target attacks, since their attacks do not involve training a model. In other words, these methods in Table 2 would only have exactly the same values as in Table 1. We did not show these methods in Table 2 for space-saving considerations.
• Taking Res-152 as the surrogate model as an example, we present the complete table here.

• Regarding the evaluation of our method on adversarially trained models in the single-target setting, we did not include this content due to considerations of the overall paper length and focus. Thank you for your suggestion, we will include this content in the final version!

4. Input Preprocessing Defense

• Thank you for your suggestion on this! We included Gaussian smoothing and JPEG compression in the main text, and included experiments on DiffPure in the appendix. The experimental results are presented in the table below, where we show the attack success rates of our method under various DiffPure $t^*$ settings and compare them with baseline methods (using ResNet-152 as the white-box model). As illustrated in the table, baseline methods are easily nullified by the purification process, whereas our method maintains a significant success rate. This further demonstrates the robustness of our approach.

• We will continue to add updated defense methods in the final version to further improve the comprehensiveness of our comparison.

5. Training Efficiency

• Thank you very much for your suggestion on this! Here we tested the training time required for the baseline methods C-GSP and CGNC on multi-target attack tasks using the same RTX 3090 GPUs as ours. Specifically, we require one day of training on two RTX 3090 GPUs, corresponding to 48 GPU hours, while this number is 50 GPU hours for C-GSP and 250 GPU hours for CGNC. This demonstrates that our model achieves superior baseline performance while maintaining excellent training efficiency. We will include this content and more detailed analysis in the final version.

6. Target Classes Selection and Method Stability Reporting

• Regarding the selection of target classes, to facilitate comparison with baselines, we follow previous work that used the NeurIPS 2017 adversarial competition dataset. Consistent with these works, we select 8 fixed target classes: 150, 426, 843, 715, 952, 507, 590, 62.
• Thank you for your suggestion on method stability evaluation! We have provided relevant content in Appendix J. We computed 95% confidence intervals (CIs) for our method. As shown in the table below, the non-overlapping intervals observed in our experiments indicate that the improvement in attack success rate achieved by our method is statistically significant with high confidence.

7. Ablation Studies Thank you for your suggestion regarding more comprehensive ablation studies! In addition to the main text, we provide some additional ablation experiments in the appendix (supplementary materials). Regarding your points:

• For the number of target classes, we will complete this experimental content in the final version according to your suggestion as much as possible.
• For sampling steps, we already have some experiments in Figure 4. Of course, we can find ways to complete this content more thoroughly. Thank you for your suggestion!
• For the effect of lower perturbation budgets. This is a very good suggestion! We chose $\epsilon=16/255$ following the default setting used in previous work on the NIPS 2017 dataset to ensure direct comparability. However, we agree that evaluating adversarial perturbations at smaller budgets is important for enhancing visual stealthiness. To address your suggestion, we conducted experiments at lower perturbation budgets for black-box targeted attacks. We used the $\epsilon=16/255$ versions of these models and clipped the generated samples to meet the $\epsilon=8/255$ and $\epsilon=4/255$ limits. For Logit and SU, we regenerated adversarial perturbations under the new budgets. The results are shown below. Values before the slash represent $\epsilon=8/255$, and values after the slash represent $\epsilon=4/255$. Our method demonstrates superior black-box transferability compared to baseline methods.

• For the suggestion of ablation experiments with more baseline methods. We will include this content in the final version as much as possible.

Thank you again for your detailed suggestions!

8. Theoretical Explanation

Thank you for your suggestion! We will further include more detailed theoretical analysis in the final version. We have some explanations in Appendix G. Additionally, we believe that the generator constrains feasible perturbations to a low-dimensional manifold (determined by network parameters), which acts as a form of strong regularization. This makes it easier to find "wide" and stable adversarial regions rather than narrow edge solutions, making it less sensitive to small changes. This helps reduce dependence on any single decision boundary shape (i.e., on the surrogate model) and improves success rates under defenses/randomization. We will add more theoretical analysis content as much as possible in the future, thank you again!

We value your insights and believe they will significantly enhance our work's clarity and impact.

Dear reviewer,

Does the response address your concerns? Could you please give a response?

Best,

AC

Thank you for your thoughtful review and detailed feedback on our paper. Below, we address the key points you raised:

Evaluation on Smaller Perturbation Budgets

• Thank you for your suggestion. Unfortunately, due to NeurIPS mechanisms this time, we are unable to further demonstrate visualization effects to you. We have conducted such tests: for different original images, we input the same target class prompt and observe the resulting adversarial perturbations. It can be seen that the perturbations are highly dependent on the structure and layout of the original image, rather than being solely determined by the target class prompt. In other words, the mechanism by which our attack works is not simply adding target class structure and semantic information to the original image.
• We chose $\epsilon=16/255$ following the default setting used in previous work on the NIPS 2017 dataset to ensure direct comparability. However, we agree that evaluating adversarial perturbations at smaller budgets is important for enhancing visual stealthiness. To address your suggestion, we conducted experiments at lower perturbation budgets for black-box targeted attacks. We used the $\epsilon=16/255$ versions of these models and clipped the generated samples to meet the $\epsilon=8/255$ and $\epsilon=4/255$ limits. For Logit and SU, we regenerated adversarial perturbations under the new budgets. The results are shown below. Values before the slash represent $\epsilon=8/255$, and values after the slash represent $\epsilon=4/255$. Our method demonstrates superior black-box transferability compared to baseline methods.

Reducing the perturbation budget makes adversarial attacks more difficult, especially for combined black-box and target attacks, which remains a challenge in adversarial research. Enhancing attack effectiveness under subtle conditions is valuable. We plan to explore this further to develop techniques that remain effective while being less perceptible to the human eye.

Transferability across CNN-based and Transformer-based Models

As you correctly pointed out, it is important to validate the transferability of our method across models with significantly different architectures. We have already provided this content in the appendix submitted as supplementary materials, specifically in Appendix E. We tested the attack success rates from Res-152 to various Transformer architecture models. We have reproduced these data in the table below, which demonstrates that our method remains effective for transfer between models with substantially different architectures.

We value your insights and believe they will significantly enhance our work's clarity and impact.

Thank you for your thoughtful review and detailed feedback on our paper. Below, we address the key points you raised:

Limited Architectural Diversity in Transfer Evaluation

• As you correctly pointed out, it is important to validate the transferability of our method across models with significantly different architectures. We have already provided this content in the appendix submitted as supplementary materials, specifically in Appendix E. We tested the attack success rates from Res-152 to various Transformer architecture models. We have reproduced these data in the table below, which demonstrates that our method remains effective for transfer between models with substantially different architectures.

• Thank you for your suggestion, we will include detailed analysis and experiments for two different surrogate models in the final version. Currently, we know that transfer between structurally similar models is easier. For example, transfer from Inc-v3 to Inc-v4 is easier compared to Res-152. We will conduct further analysis in the future.

Training Cost and Practical Scalability

• Thank you for your consideration regarding efficiency. We apologize if our description may have caused misunderstanding. In fact, the single-target variants we proposed in the paper are obtained through further fine-tuning of models that were previously trained under multi-target settings. That is to say, to obtain single-target models for all 8 classes, the actual computational cost required is 48+32=80 GPU hours, rather than only 32 GPU hours. Therefore, it cannot be said that multi-target training is more expensive than individual target models.
• This also explains why our single-target variants achieve higher attack success rates than multi-target models: if there were no performance improvement, the additional 32 GPU hours would be wasted. In other words, further fine-tuning on single targets is precisely to improve success rates. Of course, your concern about the trade-off between flexibility and performance is very meaningful. Regarding this aspect, we have some explanations in Section 4.2. Specifically, our model can still achieve performance exceeding baselines (which are all single-target models) even without further fine-tuning on individual target classes. The single-target variants are only for fair comparison settings.
• Additionally, you can observe the performance improvement that the baseline (CGNC) achieves after single-target fine-tuning. It can be seen that the performance difference between our multi-target model and single-target model is smaller. This indicates that the impact of our method's flexibility on performance is relatively small.

• Additionally, our multi-target model requires 48 GPU hours for training. Through our actual testing on the same RTX 3090, the baseline CGNC requires 250 GPU hours for training. This also demonstrates that we have lower training overhead.

No Limitation Section

Thank you for your concern about the Limitation section of our paper. In fact, our appendix is provided in the supplementary materials, and the Limitation Section is located in Appendix I.

Terminology Confusion Regarding "Victim Model"

Thank you for your very important suggestion regarding our terminology. As you correctly pointed out, since our goal is black-box attacks, it is inappropriate to refer to ($f$) in Algorithm 1 as "victim model". We will modify it to "surrogate model" in the final version.

We value your insights and believe they will significantly enhance our work's clarity and impact.