On the Adversarial Vulnerability of Label-Free Test-Time Adaptation

One of the main claims of this article is the practicability of the attacks. The paper pointed out that previous attacks like DIA need the label of the test sample. The reviewer agrees that it is very valuable to design a label-free attack. However, in Section 4, the authors claimed that the attacker has white-box access to the latest updated model parameters provided by some malicious insider. In my opinion, this assumption is impractical. First, who is the malicious insider? Can the authors provide a real-world example?

We thank you for giving us the opportunity to clarify this crucial point. Please see the common response.

In the threat model of (Cong et al., 2024), they proposed a black-box attack named TePA. TePA does not need access to the parameters of the updated model, which is more practical.

Thank you for pointing out the TePA attack. We remark that in TePA, the adversary needs to feed poisoned samples across consecutive test batches to have effect. One key distinction between our evaluation of attack's effectiveness compared to TePA is that we only consider how non-perturbed benign samples are hampered in a certain batch for evaluation. In stark opposition with TePA, we consider every new batch as an independent trial, which is a different and more challenging scenario. For example, to degrade performance using the TENT [1] on fog corruption for the CIFAR-100 dataset by only $\sim$ 2.6%, TePA needs adaptation on 40 batches (batch size of 200), with 20% of the samples poisoned. If the victim performs TTA in a episodic manner (i.e., the DNN is reset to original state), TePA will become less effective. Moreover, considering threats in our scenario is more in line with recent work [2] which strongly advocates resetting the model to prevent performance degradation.

Additionally, to evaluate TePA in our setting, we craft adversarial samples using the trained substitute black-box model provided by the authors by using the same loss function and optimization process (Diverse Input FGSM instead of PGD). We added the results in Table 1 of the updated manuscript. Table 1 shows that TePA performs nearly as well as the weakest benchmark in the paper, which is \gls{dia} with pseudo-labels (PL). Indeed, we point out that taking the model’s soft predictions as pseudo-labels is equivalent to entropy maximization with an inverted sign. Hence, maximizing entropy also maximizes deviation from the model's own predictions (pseudo-labels), resulting in similar performance for both benchmarks. The minor deviation observed is due to differences in the optimization process (e.g., iterative PGD vs. diverse input FGSM) and the surrogate models.

The authors also claimed that "existing threat models assume test-time samples will be labeled", and TePA (Cong et al., 2024) relies on access to the entire source dataset to train a substitute model. Actually, TePA also is a label-free attack, and it does not need access to the entire source dataset. It only needs a surrogate dataset that has a similar distribution of the source dataset. I suggest that the authors reorganize the introduction of related work.

Thank you for pointing this out. We have included this discussion in the Introduction section of the updated manuscript.

How many batches does FCA need to effectively decrease the target model’s performance?}}

We treat each batch as an independent trial and calculate error rates solely on unperturbed benign samples. Table 1 reports error rates for unperturbed benign samples while 20% of the batch samples are adversarially perturbed. After each adaptation step, the victim TTA state is reset to its pre-adaptation state, as if no adversarial samples were included. Thus, FCA effectively decreases the target model's performance with just a single batch.

In experiments like Table 1, this paper does not compare the attack success rate with TePA (Cong et al., 2024).}}

We have added additional results on Table 1 that compares how black-box methods fares against FCA in our evaluation settings. We did not include experiments of Imagenet-C for TePA, as for this dataset no surrogate black box model was provided by the authors.

References:

[1] Tent: Fully test-time adaptation by entropy minimization, ICLR 2021.

[2] Rdumb: A simple approach that questions our progress in continual test-time adaptation, NIPS 2024

The experiments are performed only with the ResNet architecture variants (line 316). Empirical results would benefit from including some other popular architecture e. g. one based on a vision transformer. At least in the Appendix.

Thank you for pointing this out. We have not included any Vision Transformer models in our benchmarking because the compared TTA algorithms cannot be directly applied to Vision Transformers due to the absence of batch normalization layers. The original benchmark TTA methods also did not experiment with Vision Transformer models. We attempted to adapt the layer normalization layers in Vision Transformers instead of \gls{bn} layers while keeping other aspects unchanged. However, even with a very small learning rate, adapting the parameters of layer normalization resulted in performance degradation rather than improvement. This phenomenon was observed across all three benchmark datasets and five TTA methods. Therefore, without further exploration, we cannot yet comment on the effectiveness of FCA against Vision Transformers.

We have studied the effectiveness of FCA in MobileNet architecture family and have shown the results in Table 7 of the revised manuscript. Our findings show that TTA methods exhibit similar vulnerabilities on CIFAR-10C and CIFAR-100C datasets as those observed with ResNet variants. However, for the ImageNet-C benchmark, MobileNet-v2 proved to be even more vulnerable, with performance degradation under FCA being $\sim$ 10% greater compared to the ResNet-50.

Please elaborate on how the pseudo labels are defined or add a citation (line 364)

Pseudo labels can be obtained using the current model's prediction probabilities (soft PL) or the argmax of these predictions (hard PL) for benchmarking. Both methods have similarly low efficacy on attack strength. For our experimental evaluations, we use hard pseudo labels and refer to them simply as PL for brevity.

About Typos and minor comments:

We appreciate the reviewer’s meticulous reading of our paper and the valuable suggestions. In the updated manuscript, we have incorporated all the recommendations accordingly.

Figure 1a: the process in the Figure 1a is hard to grasp for an outside reader. The diagram flow is confusing. Consider making it clearer where the flow starts and ends. The overview figure has to be more intuitive to help the reader better understand the paper idea and not to confuse them. Consider simplifying it or breaking into several Figures. The top dotted line is missing an arrow on its end.

We apologize for the previously unclear overview figure. In the revised version, we have distinctly labeled the different systems and clarified the information flow. We hope this updated figure is easier to understand.

Why is the perturbation initialized with 0.5 and not e. g. as random noise? (Algorithm 1, step 2) Wouldn’t the optimization benefit from this?

Thank you for pointing this out. In Algorithm 1, step 2, we initialized the perturbation with a value of 0.5 instead of random noise to align with the hyperparameters of the benchmark attack [1] and defense mechanism [2]. This approach ensures a direct and fair performance evaluation under equivalent conditions. However, you correctly noted that initializing with random noise could theoretically explore a broader space and potentially identify stronger adversarial perturbations. We appreciate this feedback and have run additional experiments to understand the impact of initialization. The findings of this experiment align with the suggestion, consistently creating stronger perturbations (0.04 - 0.1% error rate increase) and proving to be a superior initialization choice.

(% Error) Comparison with Different Initialization Schemes on CIFAR-100C Dataset

References:

[1] Uncovering adversarial risks of test-time adaptation, ICML 2023.

[2] Medbn: Robust test-time adaptation against malicious test sample, CVPR 2024.

To further understand how the robustness of the source DNN influences FCA, we have analyzed the performance of TTA against source DNNs known for their robustness to distribution shifts. Specifically, we have utilized the WideResNet-28 model trained with AugMix [7] from Robustbench [8]. While AugMix-trained models are effective in enhancing robustness against various distribution shifts, they remain vulnerable to attacks when deployed for TTA.

(% Error) Comparison on Robust Models

References:

[1] Uncovering adversarial risks of test-time adaptation, ICML 2023.

[2] Indiscriminate poisoning attacks on unsupervised contrastive learning, ICLR 2022.

[3] On the vulnerability of adversarially trained models against two-faced attacks, ICLR 2024.

[4] MEMO: Test Time Robustness via Adaptation and Augmentation, NIPS 2022.

[5] Theoretically principled trade-off between robustness and accuracy, ICML 2019.

[6] Robustness may be at odds with accuracy, ICLR 2019.

[7] Augmix: A simple data processing method to improve robustness and uncertainty, ICLR 2020.

[8] Robustbench: a standardized adversarial robustness benchmark, NIPS 2021.

Assumption of White-Box Access: The attack assumes white-box access to the model parameters, which might not always be realistic. Analyzing the attack's performance under more restricted settings, such as black-box or gray-box access, would provide a more complete picture of its capabilities.

Thank you for pointing this out. Please see the common response to the reviewers.

Attack Algorithm: The attack part of the FCA algorithm is almost entirely borrowed from the PGD in the adversarial sample attack, which will affect the contribution of this paper. Furthermore, does this suggest that the method could be well enhanced by using more powerful attacks?

Thank you for your remark. The main contribution of this paper is understanding how much the existing attack mechanism [1] is dependent on the availability of the true labels of test data available in an online manner in TTA settings, as well as the design of an attack mechanism based on a better understanding of the adversarial risk of TTA. We believe that using PGD as the optimization process does not weaken any of our claims or findings. PGD has been widely used to understand the security and robustness of a wide variety of learning methods and systems, including contrastive learning [2], TTA [1] and adversarial training [3]. Nevertheless, we acknowledge that the proposed method could be enhanced by using more powerful attacks or variants of PGD like having adaptive step-size and perturbation budget and we thank the reviewer for raising this point, which we plan to explore in future work.

The Robustness of Victim Model: The effectiveness of the proposed attack may be limited when targeting inherently robust models [R] or those with defense mechanisms (adversarial training), since it builds upon adversarial attack principles. How significantly does the attack's performance degrade when faced with robust or defended victim models?

Thank you for this remark. Please refer to the common response about benchmarking the effectiveness of FCA on [4].

To compare the effectiveness of TTA against adversarially-trained models, we have conducted additional experiments using models robust to both domain shift and adversarial perturbation. The results are presented in Tables 8 and 9 in the Appendix, respectively. We notice that adversarially-trained DNNs are highly effective against TTA when the same perturbation constraint$(\epsilon_{∞}=8/255)$ is used for both crafting adversarial examples and training the source \gls{dnn}. However, with a different perturbation budget, such as an $l_{2}$ norm constraint of $(\epsilon_{2}=0.5)$, TTA can still degrade performance by approximately 4%. Furthermore, for the CIFAR-100C dataset, adversarially-trained source DNNs result in more than 10% increase in error rate during adaptation with non-malicious data. This is unexpected, as TTA is generally intended to handle online data batches without adversarial perturbation, raising concerns about the robustness-utility trade-off in deploying adversarially trained DNNs.

Table: (% Error) Comparison on Adversarially Trained Models

Additionally, adversarial training is known to reduce accuracy on clean data [5,6]. Thus, further analysis is required to develop test-time defenses that are both computationally efficient yet effective against TTA without impairing TTA performance on clean or benign samples from different domains.

The number of baseline attack methods is limited. More comparison with multiple attacks is expected

Thank you for pointing this out. Understanding the security vulnerabilities of TTA is in its early stage of scrutiny, as such, the number of existing methods to compare with is limited. As pointed out by Reviewer AcHd, we have included an additional baseline TePA [1] in the revised manuscript. Should you have some specific additional baselines in mind, we would be glad to include them in the updated manuscript when the results are available.

The paper considers the worst-case scenario where the attacker has knowledge of the TTA algorithm and has white-box access to the latest updated model parameters provided by some malicious insider. Under this assumption, is it reasonable to hypothesize that the attacker has no access to the true labels?

We thank the reviewer for raising this point. Although we consider white-box access to evaluate the worst-case scenario, we assume the adversary does not have access to true labels. The rationale behind this choice is that TTA updates a DNN on unlabeled test data available in an online manner before performing inference. If true labels were available, the entire TTA setup would become redundant, allowing us to concentrate on the challenges of efficient fine-tuning. The best an adversary can do is label the data using the latest updated DNN, which we term as pseudo labels. Interestingly, even when these models are moderately accurate (with CIFAR-10C accuracy above 80%), the existing attacks proposed by [2] become almost ineffective.

References:

[1] Test-time poisoning attacks against test time adaptation models, IEEE SP 2024.

[2] Uncovering adversarial risks of test-time adaptation, ICML 2023.