Transferable Adversarial Attacks on SAM and Its Downstream Models

Dear Reviewer aXPE,

Thank you so much for taking the time to read this paper and giving constructive feedback. Please find our response below.

Give a more comprehensive analysis of the UMI noise.

Following your valuable suggestion, we conducted an evaluation to analyze the impact of the natural dataset size $N$ and the total meta iteration times $T_m$ on the UMI using the COD10K dataset. We presented the results in the following table.

The results indicate that the size of the dataset $N$ has a significant impact on the UMI. As a smaller dataset can introduce substantial bias to the generated UMI, this thereby leads to the degraded performance of the generated UMI. The effect of $T_m$ becomes minor when larger than 5.

Provide further data pertaining to $E_\Phi$ and $F_\omega ^\beta$.

We presented the performance of $E_\Phi$ and $F_\omega ^\beta$ on the COD10K dataset under five different attacks in the following table:

The results show that our method achieves the best attack performance in terms of the E-measure ($E_\Phi$) and the second-best performance on the weight F-measure $F_\omega ^\beta$. We will add those metrics in the table of our revised paper for better clarification.

Will increasing the iterative step of generating adversarial perturbation enhance the transferability?

We conducted comparative experiments under 5 different attacks with iteration times $T_a=20$ and the results are reported in Table R2 of our submitted PDF file.

The results indicate that increasing the attack iterations greatly enhances the adversarial examples on attacking camouflaged SAM, where the domain gap from the natural images is minor. However, For those tasks characterized by a substantial domain gap, increasing the iteration cannot bring performance gain.

Will the ensemble of different SAMs benefit the UMI-GRAT?

Following your valuable suggestion, we explored enhancing adversarial attacks by the ensemble under two types of scenarios:

1. The transferability from pre-trained SAM to Medical-SAM.
2. The transferability from the pre-trained SAM ViT-B to SAM ViT-H.

We adhered to the experimental settings outlined in Section 6.1 and employed an ensemble of SAM ViT-B and SAM ViT-L. We reported the mDSC for Medical SAM and mIOU for the original SAM. The results are presented in the following tables:

The results indicate that the ensemble yields performance gains in general transfer tasks while degrading the performance on the pre-trained model to fine-tuned model transfer task. As the fine-tuned model solely inherits information from its pre-trained one, incorporating uncorrelated gradient information brings unnecessary deviation, thus degrades the performance.

Expanding the scope of analysis to assess whether this adversarial threat also applies to other large foundation models.

Following your good suggestion, we evaluated the effectiveness of our proposed UMI-GRAT on two new scenarios:

1. The transferability of MUI-GRAT to other pre-trained methods. We attacked the MAE ViT-S and ViT-B that are fine-tuned on Chexpert [2] using solely the pre-trained MAE ViT-S and reported the results in Table. R3 of our submitted PDF file.
2. The effectiveness of MUI-GRAT on the general transfer attack setting. We attacked the MAE ViT-B and DenseNet-121 fine-tuned on Chexpert [2] using MAE ViT-S fine-tuned on the same dataset and reported the results in Table. R4.

We utilized models provided by [1] and followed the same experimental setting. We evaluate 8 attack methods on the Chexpert [2] dataset, where the model needs to diagnose five predefined diseases in the chest X-ray images. We reported the mean Area Under the Curve (mAUC) to evaluate the performance.

The results in Table. R3 shows that the proposed MUI-GART maintains the best method on attacking both the fine-tuned MAE-ViT-S and MAE-ViT-B models. In Table. R4, we evaluated the transferability between different ViTs and transferability from the ViT to the CNN. The experimental results indicate that our MUI-GRAT maintains effectiveness on general transfer tasks, demonstrating its good generalizability.

---

We do appreciate your constructive feedback. We will add those experiments and analyses mentioned above in the appendix of our revised version.

[1] Delving into masked autoencoders for multi-label thorax disease classification. In WACV 2023.

[2] Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In AAAI 2019.

Dear reviewer iuQV

Thank you so much for taking the time to read this paper and giving constructive feedback. Please find our response below.

Q1, Q2, and Q3: The novelty of the proposed method. The authors should discuss how their method is different from meta-learning-based approaches [2,3,4]. Clarify how their work is different from references 55 and 56. Another work[1] is missing.

Although utilizing the UAP as the initial point is discussed in [2,3,4] and downstream-agnostic attacks are discussed in [1, 55, 56], the core methodology design in our UMI is significantly different from previous work. Moreover, the proposition of GRAT in mitigating the gradient misalignment is unique, novel, and effective. The main differences are:

1. Compared with meta-learning approaches: [3] utilizes the UAP to generate AEs with a one-step update, and [4] uses UAP for adversarial training. [2] enhances the existing UAP under the few-shot learning scenarios: learning a UAP with a few examples to attack the same victim model. Different from previous work, we first utilize the UMI to extract the intrinsic vulnerability inherent in the pre-trained foundation model and utilize it as the prior knowledge to enhance the attack on fine-tuned downstream models.

2. Compared with model-agnostic approaches: [1] first proposed a representation-based adversarial attack to enhance the downstream-agnostic adversarial training. [55] proposed the first framework for generating downstream-agnostic UAP on self-learning models and [56] extends this attack to multimodality. However, those methods do not consider the utilization of intrinsic vulnerability in the pre-trained model and the gradient misalignment brought by fine-tuning, thus do not work well when the downstream dataset exhibits a distinctive domain gap from the pre-trained dataset (e.g. from a natural dataset to the medical dataset).

The novelty of our work is two-fold:

1. Exploitation of intrinsic vulnerability via UMI: we first utilize the UMI to extract the intrinsic vulnerability inherent in the pre-trained foundation model and utilize it to enhance the attack on fine-tuned downstream models.

2. Rectification of gradient misalignment via GRAT: Inspired by our Proposition 1, which formulates the deviation occurred on attacking the unknown fine-tuned model, we propose the GRAT to effectively mitigate this misalignment. The experimental results in Figure 4 and Table 2 demonstrate that when the fine-tuned model has a pronounced gradient update, the proposed GRAT greatly rectifies the deviation and brings a great performance gain.

We will discuss and cite all the mentioned work above in our revised paper.

Q2 and Q6: Why compare with intermediate-level feature-based approaches? They should also compare it with different downstream agnostic approaches.

The Reason for comparison with ILPD is: as shown in Equation (7) of the submitted paper, our attack aims to maximize the feature distance. We thus discuss and compare our method with the SOTA feature-based attack.

Comparison with [1]: according to Section 3.1 and Equation (4) in [1], the attack of [1] directly perturbs the input to maximize feature distortion via MI-FGSM. According to Equation (7) of the submitted paper, all methods compared in Table 1 aim to maximize the feature distortion of SAM's image encoder. This means the compared MI-FGSM in Table 1 is totally the same as the attack algorithm proposed in [1]. We will add more explanations in our revised version to dispel this confusion.

Comparison with [55, 56]: as [55, 56] focus on universal adversarial perturbation, directly comparing them with the input-specific attacks is somewhat unfair to them. We evaluated the Adv-PER [55] in attacking normally trained and adversarial-trained Medical SAMs and reported the experimental results in Table R5 of our submitted PDF file, indicating that the Adv-PER does not work well when the downstream dataset exhibits a significant domain gap from the pre-trained dataset.

Q4. Clarify the claim "without accessing the downstream task".

Our attack perturbs the feature encoder of SAM, which, by fine-tuning the decoder, can zero-shot and few-shot transfer to various downstream tasks, such as edge prediction. Without accessing the task, the distortion in the feature encoder will adversely affect the performance across different types of decoders. To substantiate this and demonstrate the generation, we conducted a new experiment on attacking the MAE model fine-tuned on the Chexpert [5], a task for diagnosing chest X-ray diseases. We reported the results in Table R3 and R4 of our submitted PDF, demonstrating the effectiveness and generalization of our method across different downstream tasks.

Q5. The authors should include some specific aspects of SAM to make their attack more unique.

As stated in lines 548-549 in our submitted paper, our proposed UMI-GRAT is not contingent upon a prior on model’s architecture, suggesting its potential applicability across various model paradigms. The experiments on attacking the fine-tuned MAE ViTs and CNNs in Tables R3 and R4 demonstrate its good generalization. Following your suggestion, we will clarify this in our revised version.

Q7. How to choose the threshold lambda in Equation 9?

Thanks for your good suggestion, we will add the discussion below in our revised paper:

"We initialize the $\lambda$ with a value of 0.05, which is easily satisfied by most inputs, at the first epoch and increase it by a factor of 2 if at least 50% of the inputs meet this threshold."

---

We greatly appreciate your constructive feedback and will add all the experiments with analyses and cite all mentioned papers in our revised version.

[5] Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In AAAI 2019.

Dear Reviewer Q28B,

Thank you so much for taking the time to read this paper and giving constructive feedback. Please find our response below.

1. More experiment settings (e.g. against pretrained MAE models) are warranted to demonstrate the generalizability.

Following your good suggestion, we evaluated the effectiveness of our proposed UMI-GRAT on two new scenarios:

1. The transferability of MUI-GRAT to other pre-trained methods. We attacked the MAE ViT-S and ViT-B that are fine-tuned on Chexpert [2] using solely the pre-trained MAE ViT-S and reported the results in Table. R3 of our submitted PDF file.
2. The effectiveness of MUI-GRAT on the general transfer attack setting. We attacked the MAE ViT-B and DenseNet-121 fine-tuned on Chexpert [2] using MAE ViT-S fine-tuned on the same dataset and reported the results in Table. R4.

We utilized models provided by [1] and followed the same experimental setting. In both scenarios, we evaluate 8 attack methods on the Chexpert [2] dataset, where the model needs to diagnose five predefined diseases in the chest X-ray images. Following [1], we report the mean Area Under the Curve (mAUC) to evaluate the performance. Each experiment is run for 5 times.

The results in Table. R3 shows that the proposed MUI-GART maintains the best method on attacking both the MAE-ViT-S and MAE-ViT-B models fine-tuned on the downstream data. Moreover, the combination of the MUI-GRAT with the second-best method BSR [3] brings a further enhancement to the performance.

In Table. R4, we evaluated the transferability between different ViTs and transferability from the ViT to the CNN. The experimental results indicate that our MUI-GRAT maintains effectiveness on general transfer tasks (MUI-GRAT achieves the second-best attack performance compared to other SOTA methods), demonstrating its good generalizability.

2. How does the domain gap between the natural image dataset used to obtain the universal adversarial trigger and the downstream dataset influence the effectiveness of the attack?

The efficacy of the universal adversarial trigger increases as the domain gap between the downstream dataset and the natural image dataset narrows. As mentioned in lines "334-340" of the submitted paper, the UMI extracts the intrinsic vulnerability inherent in the foundation model, thus being more effective when the victim model inherits significant information from the pre-trained model. The following table presents the performance of the SAM before and after being fine-tuned on downstream tasks:

The results show that the fine-tuning brings a significant performance gain for Medical SAM but a minor performance gain for Camouflaged SAM, indicating that the domain gap between the medical and natural datasets is huge and the model is accompanied by a pronounced gradient update. The domain gap between the camouflaged and natural datasets is small so that the gradient modification following fine-tuning is minor. The performance gain shown in the rest data demonstrates that the UMI derives a greater advantage than the GRAT when the domain gap is small and GRAT displays a contrary trend, which aligns with our analysis in Section 4.

3. How effective is the proposed method against adaptive defense?

Following your constructive feedback, we conduct comparative experiments on attacking the adversarial trained Medical SAM. As our attacks aim to maximize the feature distance, we thereby consider two types of adversarial training (AT) mechanisms:

1. Feature-wise AT: the defender is aware of feature-wise attacks, thus assuming an attacker whose objective is to maximize the distance of feature embedding during AT, and hence the defender minimizes the adversarial loss.
2. Output-wise AT: the defender is unaware of feature-wise attacks, thus assuming an attacker whose objective is to maximize the final segmentation loss during AT, and hence the defender minimizes the adversarial segmentation loss.

We optimize the Medical SAM by incorporating the adversarial loss into the training loss with a weight hyperparameter $\tau$. We evaluate $\tau=0.1\,0.5$ and use the MI-FGSM with iteration $T_a =1, 5$ with bound $\epsilon=10$ as the attacking strategy.

We ran 9 different attacks on 6 different AT models on CT-Scan and presented the results in Table R.5. The results demonstrate that:

1. The proposed UMI-GRAT remains the most effective method even if the model has a certain robustness via AT, demonstrating the effectiveness of our proposed UMI-GRAT.
2. Feature-wise AT surpasses output-wise adversarial training for those feature maximizing attacks in most scenarios.
3. The robustness of AT varies drastically with different training hypermeters. Enlarging the weight $\tau$ benefits the robustness. An intriguing finding is that using a stronger gradient attack (e.g . MI-FGSM $T_a =1\,to \, 5$) during AT may damage the robustness towards adversarial examples generated from the pre-trained model.

---

We greatly appreciate your constructive feedbacks. We will add those experiments and analyses mentioned above in the appendix of our revised version.

[1] Delving into masked autoencoders for multi-label thorax disease classification. In WACV 2023.

[2] Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In AAAI 2019.

Dear Reviewer,

Thank you so much for your follow-up. The PDF file is attached to the overall Author Rebuttal. However, it seems that there is a bug in the OpenReview website that makes this rebuttal invisible to reviewers. We believe that NeurIPS will address this issue soon, and you can find the submitted PDF in the Author Rebuttal section.

Best regards, The Authors

Dear Reviewer Qxcd,

Thank you so much for taking the time to read this paper and giving constructive feedback. Please find our response below.

1. How does the performance of UMI-GRAT vary with different hyperparameters, such as the bound $\epsilon$ and iterations?

Following your good suggestion, we conducted experiments using two additional sets of hyperparameters: bound $\epsilon=4$ with iteration $T_a=10$ and bound $\epsilon=10$ with iteration $T_a=20$. We evaluated 5 different attacks and reported the experimental results in Table R2 of our submitted PDF file. The results indicate that:

1. In datasets that exhibit a large domain gap from natural image datasets (e.g., medical and shadow datasets), the attack bound $\epsilon$ is critical for transferability. Reducing the bound leads to a substantial performance decline for all attack algorithms. For the camouflaged object segmentation that shares a small domain gap with the original SAM's task, reducing the norm bound only causes a marginal performance drop.
2. Increasing the attack iterations markedly enhances the AEs when the surrogate and victim domains are proximate. However, for those tasks characterized by a substantial domain gap, increasing the iteration has little help.

2. Is the proposed method a universal transfer attack method? Can the UMI-GRAT and SOTA be compared under a general transfer attack test setting?

Following your good suggestion, we evaluated the effectiveness of our proposed UMI-GRAT on two new scenarios:

1. The transferability of MUI-GRAT to other pre-trained methods. We attacked the MAE ViT-S and ViT-B that are fine-tuned on Chexpert [2] using solely the pre-trained MAE ViT-S and reported the results in Table. R3 of our submitted PDF file.
2. The effectiveness of MUI-GRAT on the general transfer attack setting. We attacked the MAE ViT-B and MAE DenseNet-121 fine-tuned on Chexpert [2] using MAE ViT-S fine-tuned on the same dataset and reported the results in Table. R4.

We utilized models provided by [1] and followed the same experimental setting. In both scenarios, we evaluate 8 attack methods on the Chexpert [2] dataset, where the model needs to diagnose five predefined diseases in the chest X-ray images. We reported the mean Area Under the Curve (mAUC) to evaluate the performance.

The results in Table R3 and R4 demonstrate the great generalizability of our proposed method. Table R3 shows that the proposed MUI-GART maintains the best method on attacking both the fine-tuned MAE-ViT-S and MAE-ViT-B models. In Table. R4, the evaluation of the transferability between different ViTs and transferability from the ViT to the CNN indicates that our MUI-GRAT maintains effectiveness (the second-best method) on general transfer tasks.

3. How does it compare with NCS [R2], ANDA [R3], DeCowA [R4] and L2T [R5]

Following your good suggestion, we evaluated two currently open-sourced methods, ANDA[R3] and L2T[R5]. We conducted experiments considering the above two scenarios along with attacking the normally trained and adversarial trained Medical SAM. We set the n_ens=5 for ANDA and num_ scale=3 for L2T and keep the rest hyperparameters the same. We reported the experimental results in Table R3, R4, and R5 in our submitted PDF file.

The results show that L2T performs well in both transfer tasks on MAE-based models while failing in attacking the SAM and adversarial-trained SAM. ANDA performs well on the general transfer tasks while failing on all pre-trained to fine-tuned transfer tasks. We hypothesize the reason for their failures as:

1. Due to the great misalignment in the gradient discussed in Definition 1 and Proposition 1, the transfer between pre-trained and fine-tuned models is much harder than the general one.
2. The feature-oriented attacking task may potentially undermine those output-oriented attack methods.

4. How does the proposed method perform when the model has a certain robustness (such as adversarial training)?

Following your constructive feedback, we conducted comparative experiments on attacking the adversarial trained Medical SAM. As our attacks aim to maximize the feature distance, we thereby consider two types of adversarial training (AT) mechanisms:

1. Feature-wise AT: the defender is aware of feature-wise attacks, thus assuming an attacker to maximize the distance of feature embedding during AT, and the defender thereby minimizes the adversarial loss.
2. Output-wise AT: the defender is unaware of feature-wise attacks, thus assuming an attacker to maximize the final segmentation loss during AT, and the defender thereby minimizes the adversarial segmentation loss.

We optimize the Medical SAM by incorporating the adversarial loss into the training loss with a weight factor $\tau$. We used $\tau=0.1\,0.5$ and took the MI-FGSM with iteration $T_a =1, 5$ and the same bound $\epsilon=10$ as the attacking strategy.

We ran 9 different attacks on 6 different AT models and presented the results in Table R.5, demonstrating that:

1. The proposed UMI-GRAT remains the most effective method even if the model has a certain robustness via AT, demonstrating the effectiveness of our proposed method.
2. Feature-wise AT surpasses output-wise AT towards those feature maximizing attacks in most scenarios.
3. Enlarging the weight $\tau$ benefits the robustness. An intriguing finding is that using a stronger gradient attack during AT (e.g . 5-steps MI-FGSM) may damage the robustness.

---

We greatly appreciate your constructive feedback and will add all those experiments with analyses and cite all mentioned papers in our revised version.

[1] Delving into masked autoencoders for multi-label thorax disease classification. In WACV 2023.

[2] Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In AAAI 2019.

Dear Reviewer pDnz,

Thank you so much for taking the time to read this paper and giving constructive feedback. Please find our response below.

1.The readability of the Methodology section of this article is somewhat poor. The authors define the problem to be solved through the form of propositions. Similarly, if the authors could summarise the formulas as Theorem and put the proof process (both formulas and reasoning) specifically in the supplementary material, it would make the article more coherent.

Thanks for your constructive feedback. We will consolidate the Proposition and Equations (15) through (18) into a Theorem and put the remaining formulas and explanatory text in the appendix.

The randomness of the experimental results is unknown. It would be better to report a set of randomness experiments on a smaller dataset and simpler settings.

Following your good suggestion, we evaluated 10 attack methods presented in our paper over 5 random seed runs on the subset of SAM's downstream tasks and reported the mean performance with its standard deviation. We use the same experimental setting provided in Section 6.1 of our submitted paper.

The details of each subset are:

1. For the Medical SAM, we selected 'case0008' (comprising 148 images with a resolution of 512x512) from the validation set of the CT-Scan dataset.
2. For the Shadow SAM, we randomly selected 100 images with a resolution of 1024x1024 from the test set of the ISTD dataset.
3. For the Camouflaged SAM that is evaluated across three different datasets, we randomly selected 40 images with a resolution of 1024x1024 from each dataset (COD10k, CAMO, CHAME), with a total of 120 images.

The results are shown in Table R1 of our submitted PDF file, which demonstrates that the randomness is small and similar among all attacking methods. The uncertainty level of UMI-GRAT in mean Hausdorff Distance (mHD) is marginally higher compared to the other methods. This can account for the higher mHD value achieved by the UMI-GRAT.

We greatly appreciate your constructive feedback. We will conduct the randomness evaluation over the entire dataset and update the results in Table 1 and Table 2 of our paper in the revised version.