On the Adversarial Risk of Test Time Adaptation: An Investigation into Realistic Test-Time Data Poisoning

We highly appreciate the reviewer's efforts in providing valuable comments and constructive suggestions for improvement on our submission.

W1: Distinction between RTTDP and TePA in Table 1.

We would like to highlight the key differences between our proposed RTTDP and TePA protocols:

• Attack Order: TePA follows an offline attack order, where poisoned data are injected into the TTA model before any benign test data. In contrast, RTTDP follows an online attack order, where poisoned data can be injected dynamically between batches of benign test data.

• Adversary Knowledge: TePA assumes that the adversary has access to the model parameters when generating poisoned data, while RTTDP assumes that the adversary cannot access the real-time model parameters.

• Timing of Poisoned Data Injection: RTTDP allows poisoned data to be injected at any time in the test data stream, making it compatible with real-time online scenarios. TePA, on the other hand, requires poisoned data to be introduced prior to any benign user data.

• Test Domain Setting: RTTDP operates under the continual test-time adaptation (CoTTA) [1] setting, where the test data distribution changes gradually over time. However, TePA evaluates the poisoning effect under an individual test domain adaptation, which assumes a static and single test data distribution.

[1] Continual Test-Time Domain Adaptation

W2(A): Inconsistent comparison between poisoning and adversarial attacks.

Thank you for your valuable feedback. We would like to clarify that in Tables 2 to 4 of the manuscript, all competing attack objectives, including DIA, TePA, and MaxCE, are implemented within the RTTDP protocol. All these methods generate poisoned data using the PGD attack. However, due to the lack of access to real-time model parameters under the RTTDP protocol, we use the source model as the threat model for comparison.

Here are the specifics of the implemented methods:

• DIA: We adapt the original DIA objective to our RTTDP framework by splitting $\mathcal{B} _{ab}$ into two equal subsets, $\mathcal{B} _{ab}^p$ and $\mathcal{B} _{ab}^b$, with a 1:1 ratio. We then generate poisoned samples $\mathcal{B} _a^p$ by maximizing the cross-entropy loss of $\mathcal{B} _{ab}^b$, as follows:

  $\mathcal{B} _a^p = \arg\min _{\mathcal{B} _a^p} E _{(x,y)\in\mathcal{B} _{ab}^b} \left[-\text{CrossEntropyLoss}\left(f(x, \theta _{\text{source}}(\mathcal{B} _{ab}^b \cup \mathcal{B} _a^p)), y\right)\right]$

• TePA: Following the original paper, we generate poisoned samples by maximizing the entropy of the samples, expressed as:

  $\mathcal{B} _a = \arg\min _{\mathcal{B} _a} E _{(x,y)\in\mathcal{B} _a} \left[-\text{Entropy}\left(f(x, \theta _{\text{source}}(\mathcal{B} _a))\right)\right]$

• MaxCE: We generate poisoned samples by maximizing the cross-entropy loss on the poisoned samples, as follows.PGD attack is employed to attack this loss and we are sorry for referring to the wrong reference (FGSM).

  $\mathcal{B} _a = \arg\min _{\mathcal{B} _a} E _{(x,y)\in\mathcal{B} _a} \left[-\text{CrossEntropyLoss}\left(f(x, \theta _{\text{source}}(\mathcal{B} _a)), y\right)\right]$

In terms of the optimization strategy, we employ the PGD attack, as detailed at line 342 in the manuscript. After generating the poisoned samples, they are injected into the TTA pipeline alongside the normal users' samples $\mathcal{B} _b$. The attack success rate is then evaluated on the benign users' samples.

We will ensure this explanation is clarified in the Benchmark Poisoning Methods subsection of the revised manuscript. This should resolve any confusion between the different attack strategies and their implementation under RTTDP.

Beyond using PGD attack, we also evaluated AutoAttack to generate strong adversarial samples with details presented in responts to W3&Q3. Despite being strong as adversarial samples, samples generated by AutoAttack do not pose poisoning effect to TTA model.

W2(B): The further descriptions of Realistic Test-Time Data Poisoning.

Traditional data poisoning is typically an offline process, where the poisoned dataset is used to train a randomly initialized model over multiple epochs until convergence. In contrast, our proposed RTTDP introduces a novel online test-time data poisoning setting.

RTTDP operates within a Test-Time Adaptation (TTA) [2] framework, which is an online fine-tuning method. TTA methods adapt a source pre-trained model to the test data distribution. Specifically, when a testing data batch is fed into the TTA model, it updates the online model using an unsupervised loss for a few iterations (usually one) with the current testing data batch. The model then instantly outputs predictions for this batch. In RTTDP, poisoned data are directly injected into the TTA model during the test phase, rather than being handled separately offline. These poisoned samples are treated similarly to normal testing batches and are used to update the online model for a single iteration. The objective is to investigate how these poisoned data affect the predictions of other benign samples.

To improve clarity, we will elaborate on these points in the revised manuscript. This includes adding more details about the preliminary concepts, such as an introduction to Test-Time Adaptation and a comparison with Traditional Data Poisoning approaches. More details about the comparison with traditional data poisoning can be found in the response to Reviewer AP94 #W1.

[2] Tent: Fully Test-time Adaptation by Entropy Minimization

W3&Q3: Not impactful results.

Thanks for your suggestions. We first would like to clarify that the MaxCE used into our experiments is also optimized via PGD attack and we apologize for making reference to the earlier paper in the original manuscript, where FGSM was proposed. The improvement based on MaxCE attack method is not marginal, especially for TENT, EATA and SAR. Furthermore, we supplement the experiment using a more advance adversarial attack method, i.e. AutoAttack, to generate the poisoned samples. The results on CIFAR10-C and ImageNet-C are shown below, and we make the following observations.

• The goals of adversarial attack methods and data poisoning methods are significantly different. The adversarial attack methods aim to make their own samples predict incorrectly by corrupting the test samples with adversarial noise. However, data poisoning methods aim to inject some poisoned samples so that the online model updated on them would perform poorly on other benign samples.
• AutoAttack, as a more advanced adversarial attack method, would produce worse results than MaxCE-PGD. We used code from the official AutoAttack repository and repeated the experiments to ensure the reliability and reproducibility of these results.

We will supplement all the results of AutoAttack on the CIFAR10/100-C and ImageNet-C, as well as more implementation details about MaxCE in the camera-ready version.

The results on CIFAR10-C dataset with Uniform attack frequency.

The results on ImageNet-C dataset with Uniform attack frequency.

Why do we include MaxCE in our comparisons?

This is because we want to see if the adversarial effect can be transferred from the poisoned samples to the benign users' samples. In our experiment, we also compare with some common adversarial attack methods, such as MaxCE-PGD. For the adversarial attack method, we use it to generate the poisoned samples instead of directly modifying the samples of the benign users to achieve realism in the RTTDP protocol. Through comparisons, we observe that (i) the goal of adversarial attack is very different from that of our data poisoning; (ii) Figure 2(b) found that the samples generated by directly maximizing cross-entropy would produce a significant bias with the normal samples. This observation largely motivates us to introduce a feature constraint to transfer the adversarial effect from the poisoned samples to the other benign samples. Although some of the results in ImageNet-C may behave differently than we would have expected, we analyse that because the source results in ImageNet-C are so bad, the feature points are so dispersed in the feature space that it is difficult to approximate them by a Gaussian distribution.

W5&Q2: Confusing experimental setup.

Thank you for raising this concern. We would provide a clearer explanation of the design choices in our experimental setup for "Benchmarking Poisoning Methods." Below, we revisit the experimental setups of the original methods, explain the differences under our RTTDP protocol, and justify the adaptations we made to ensure fair comparisons. Additionally, to enhance reproducibility, we promise that we will release all source code, including our method and the competing methods, as soon as this manuscript will be accepted.

Commonalities among the different protocols:

The three protocols, TePA, DIA, and RTTDP, share several overarching goals and assumptions:

• All protocols aim to evaluate the adversarial risks posed to Test-Time Adaptation (TTA) by injecting poisoned samples into the test data stream.
• All protocols allow the adversary to obtain the source model, since the source model is usually the well-known pre-trained model, e.g., ImageNet pre-trained ResNet, and the open-source foundation model, e.g., DINOv2, SAM.

Key Differences Between Protocols:

Despite these commonalities, the protocols diverge significantly in their attack setups, as summarized below:

1. TePA Protocol:

   • Poisoning Objective: TePA generates poisoned samples by maximizing the entropy of its own crafted samples with respect to the source model’s predictions.
   • Injection Strategy: All poisoned samples are injected into the TTA pipeline before any benign users’ samples are processed. This approach simulates an offline attack, which is unrealistic in real-world scenarios where the adversary cannot fully control the sequence of test samples in advance.

2. DIA Protocol:

   • Poisoning Objective: DIA optimizes poisoned samples to maximize the cross-entropy loss of other users’ benign samples. And DIA relies on direct access to online model parameters and the ability to observe other benign users’ samples.
   • Injection Strategy: Poisoned samples are injected into the TTA pipeline alongside corresponding benign users’ samples.
   • Limitations for Realistic Settings:
     • Online Model Access: In practice, the adversary cannot access or modify the parameters of the online model.
     • Other Benign Users’ Data: The adversary is unlikely to observe benign users’ samples, let alone validation samples required for optimizing poisoning objectives.

3. RTTDP Protocol (Ours):

   • Poisoning Objective: RTTDP uses a surrogate model (initialized as the source model) to generate poisoned samples. This surrogate model is updated iteratively based on the feedback from previously injected poisoned samples.
   • Injection Strategy: Poisoned samples are injected into the TTA pipeline either uniformly or non-uniformly:
     • Uniform: Poisoned minibatches are evenly distributed across the test data stream.
     • Non-Uniform: Poisoned minibatches are concentrated at specific points in the test data stream.
   • Realistic Assumptions: RTTDP assumes the adversary has no access to the online model parameters or other users’ samples. This ensures a more realistic and practical attack scenario.

Adaptations of Competing Methods for RTTDP:

To ensure fair comparisons under RTTDP protocol, we made the following adjustments to the competing methods:

1. TePA: We preserved TePA’s original poisoning objective but adapted the injection strategy to follow the Uniform or Non-Uniform attack frequency in RTTDP protocols.

2. DIA:

   • Replacing Online Model Parameters: DIA’s original objective relies on online model parameters, which are inaccessible in RTTDP. We replaced these parameters with the source model parameters.
   • No Access to Benign Users' Samples for Optimization: DIA uses benign users’ samples as optimization targets in its original setup. To meet RTTDP’s constraints, we split $\mathcal{B} _{ab}$ into two equal subsets, $\mathcal{B} _{ab}^p$ and $\mathcal{B} _{ab}^b$, with a 1:1 ratio. We then generate poisoned samples $\mathcal{B} _a^p$ by maximizing the cross-entropy loss of $\mathcal{B} _{ab}^b$. Details of this modification are provided in our response to W2(A).

Justification of Design Choices:

• Realism: RTTDP reflects more realistic assumptions than TePA and DIA by removing unrealistic assumptions, such as online model access and visibility of other users’ samples.
• Fairness: All competing methods are implemented under the RTTDP protocol, with necessary adaptations to maintain methodological integrity while adhering to RTTDP constraints.
• Reproducibility: We will release the complete implementation of our proposed method and the adapted competing methods to ensure full transparency and reproducibility.

W4&Q1: Unsupported and unclear methodology.

We appreciate the reviewer’s concerns and provide a detailed explanation below to clarify our methodology, specifically the transition from a bilevel to a single-level optimization and the rationale for our attack objective.

Transition to Single-Level Optimization

The original bilevel optimization involves an inner loop where the model adapts to test samples, including poisoned and benign samples, and an outer loop to optimize the attack objective. This structure is computationally intensive and impractical under the constraints of the RTTDP setting. To address this, we adopt the approximation strategy used in DIA [3] and make two key adjustments:

1. Discarding the Inner Optimization:

   • In DIA [3], the inner optimization is approximated by assuming $\theta_t^* \approx \theta_t$, where $\theta_t^*$ represents the parameters after a full adaptation step, and $\theta_t$ represents the current parameters.
   • This approximation is justified as TTA models typically update minimally during a single minibatch iteration, resulting in minor perturbations to $\theta_t$. Thus, the approximation retains practical relevance while simplifying the problem.

2. Surrogate Model for Online Parameters:

   In the RTTDP protocol, direct access to online model parameters $\theta_t$ is unrealistic. Instead, we replace $\theta_t$ with the surrogate model parameters $\hat{\theta}_t$, which are accessible and trained to approximate the online model’s behavior.

3. Final Optimization Objective:

   After these adjustments, the optimization simplifies to a single-level objective, as shown in Eq. 4 of the manuscript. This formulation allows efficient generation of poisoned samples while adhering to the realistic constraints of RTTDP.

Additionally, we derive the in-distribution attack objective:

Our proposed attack leverages the dependence of TTA models on self-training mechanisms, which aim to maximize confidence on pseudo-labels for adaptation. The core idea is as follows:

1. Crafting Poisoned Samples:

   • The poisoned samples are constrained to maintain the shallow feature distribution of benign samples, satisfying $D(P_a, P_{ab})=0$ where $P_a$ and $P_{ab}$ denote the shallow feature distributions of poisoned and benign samples, respectively.
   • However, these samples are intentionally manipulated to induce incorrect predictions by $\mathcal{L}_{atk}$. The model perceives these samples as valid but reinforces erroneous patterns during self-training.

2. Reinforcing Erroneous Information: When the TTA model adapts to poisoned samples, it learns and reinforces incorrect associations. This creates a vulnerability, as future test samples with similar shallow feature distributions are more likely to be misclassified by the online model.

3. Exploiting TTA Dependence on Test Data: Since TTA methods iteratively adapt using incoming test samples, our approach leverages this dependency to propagate the error induced by poisoned samples throughout the adaptation process.

[3] Uncovering Adversarial Risks of Test-Time Adaptation

W6: Missing details on attack queries.

Under our RTTDP protocol, all methods generate the poisoned data based on the offline model (source model or surrogate model), and they all use the 40-iterations PGD attack as the optimized strategy (Please refer to the line 339 to 342 in the manuscript, different methods only change the optimized objective/loss). Each poisoned data queries the online model (TTA model) only for one time to obtain the predictions, like the normal users' samples.

MP1: The asymmetry KLD.

Thanks for your suggestions. We would clarify clearly about the used asymmetry KLD in the revised manuscript. We leverage the symmetry KLD loss in Eq. 1 which would provide a more robust and smooth aligning process between two distributions.

MP2&3&4: The improvement on Equations, Figures and Background introduction.

Thanks for your suggestions. We would improve the readability and conciseness of the equations, Figure 1 and the background introduction in our revised manuscript.

---

Thank you for your valuable comments. We hope the above clarifications and additions comprehensively address your concerns. Your insights have been instrumental in improving the quality of our manuscript. We sincerely appreciate your support and constructive suggestions!

Best regards,

The Authors

W1: Comparison between our proposed method and existing works [1,2].

Thank you for your suggestion. We will discuss and compare the two papers [1,2] in the revised manuscript. These papers focus on traditional data poisoning task, which differ significantly from the RTTDP setting we propose. Below, we outline the key differences in the goals, theoretical definitions, and corresponding methods between the traditional data poisoning and RTTDP tasks.

The Goal:

• Traditional Data Poisoning (DP) [1,2] aims to poison all the training data used to train a model from scratch under a fully supervised protocol until convergence, in order to perform poorly on normal test samples.

• Our proposed RTTDP investigates the robustness of TTA methods against injecting some poisoned data into the test data stream. Therefore, it first follows the TTA pipeline where the online model is initialized as a source pre-trained model, and the TTA model uses each minibatch of test data to update the online model via an unsupervised TTA loss, and instantly produces predictions for the current test data. In RTTDP, the poisoned samples are injected into the TTA pipeline like normal test samples, and the performance is measured on benign users' (normal) samples.

Theoretical Definitions:

• In line with [1,2], we define traditional data poisoning as follows, where $\mathcal{T}_{te}$ and $\mathcal{T}\_{tr}$ denote the test and training sets, and $\mathcal{L}$ usually represents the cross-entropy loss:

  $\hat{\epsilon} = \arg\max_{\epsilon}\sum_{(x_i,y_i)\in\mathcal{T}_{te}} \mathcal{L}(f(x_i;\theta(\epsilon)),y_i)$

  $\text{s.t.} \quad \theta(\epsilon) = \arg\min_\theta \sum_{(x_i,y_i)\in\mathcal{T}_{tr}} \mathcal{L}(f(x_i + \epsilon_i; \theta), y_i)$

• Our RTTDP follows a DIA optimization objective, where $\mathcal{B} _a$ and $\mathcal{B} _b$ represent poisoned and benign minibatch data, $\mathcal{L} _{atk}$ is the attack loss (e.g., maximizing cross-entropy), and $\mathcal{L} _{tta}$ is the unsupervised loss for TTA:

  $\mathcal{B} _a = \arg\min _{\mathcal{B} _a} \sum _{(x,y)\in\mathcal{B} _b} \mathcal{L} _{atk}(f(x;\theta _t^*(\mathcal{B} _a \cup \mathcal{B} _b)), y)$

  $\text{s.t.} \quad \theta _t^*(\mathcal{B} _a \cup \mathcal{B} _b) = \arg\min _\theta \mathcal{L} _{tta}(f(\mathcal{B} _a \cup \mathcal{B} _b;\theta _t))$

  However, this formula presents challenges:

  • Inner optimization challenge: The TTA loss $\mathcal{L} _{tta}$ and the online model $f(x;\theta)$ do not allow backward gradients to the poisoned samples $\mathcal{B} _a$. Moreover, black-box attack methods, which rely on repeated queries for gradient estimation, are impractical since each batch of poisoned samples requires numerous queries, and the online model is updated after each query.
  • Outer optimization challenge: In RTTDP, we cannot observe benign user samples when generating poisoned samples. This is equivalent to not having access to $\mathcal{T}_{te}$ in the traditional data poisoning problem. Thus, directly targeting validation data would not effectively evaluate the risk of poisoned samples in the TTA model.

• To address the inner optimization problem, we approximate $\theta _t^* \approx \theta _t$ since each minibatch updates the online model only slightly. The objective then becomes:

  $\mathcal{B} _a = \arg\min _{\mathcal{B} _a} \sum _{(x,y)\in\mathcal{B} _b} \mathcal{L} _{atk}(f(x;\theta _t(\mathcal{B} _a \cup \mathcal{B} _b)), y)$

  where $\theta(\mathcal{B} _a\cup\mathcal{B} _b)$ indicates forwarding $\mathcal{B} _a\cup\mathcal{B} _b$ to update the BN statistics. This objective would lead to a trivial solution that $\mathcal{B} _a$ is effective only for the current $\mathcal{B} _b$ data through easily introducing biased normalization in each BN layer, and it has little effect while $\mathcal{B} _a$ and $\mathcal{B} _b$ are in seperated batch. Furthermore, the adversary cannot observe $\mathcal{B} _b$ in practice.

• To address the outer optimization problem, we modify the objective function using the PAC learning framework that constrains the shallow feature distributions of $\mathcal{B} _a$ and $\mathcal{B} _{ab}$ are similar, as shown in the manuscript (Eq. 2-4). The final objective becomes:

  $\mathcal{B} _a = \arg\min _{\mathcal{B} _a} \sum _{(x,y)\in\mathcal{B} _a} \mathcal{L} _{atk}(f(x;\hat{\theta} _t(\mathcal{B} _a)), y), \quad \text{s.t.} \ D(P _a, P _{ab}) = 0$

  where $\hat{\theta}_t$ is the surrogate model used due to the inaccessibility of the online model $\theta_t$.

Q3(B): Is this just the success of standard adversarial attacks?

No, they are not the results of standard adversarial attacks. They are the results of the source pre-trained model directly validated on the test data stream as the traiditional testing. Because our RTTDP don't allow the adversary directly observe the benign samples that are used as the validation samples, so we don't have the results of standard adversarial attacks on benign samples. However, we can test the performance of standard adversarial attacks on our RTTDP setting by generating the poisoned samples using standard adversarial attacks and injecting them into the TTA pipeline to update the online model, and the results are still evaluated on the benign samples, as the results of MaxCE-PGD, GMSA-AVG, GMSA-MIN, AutoAttack in this rebuttal.

---

Thank you for your valuable comments. We hope the above clarifications and additions comprehensively address your concerns. Your insights have been instrumental in improving the quality of our manuscript. We sincerely appreciate your support and constructive suggestions!

Best regards,

The Authors

Corresponding Methods:

• Traditional Data Poisoning:

  • [1] generates adversarial noise by minimizing the cross-entropy loss of training samples on a randomly initialized model. The poisoned samples produced with this noise prevent a randomly initialized model from learning true semantic information.
  • [2] uses a pre-trained model to generate noise that confuses model predictions, i.e., $\hat{\epsilon} = \arg\min _{\epsilon} \sum _{(x _i,y _i)\in\mathcal{T}} \mathcal{L}(f(x _i + \epsilon _i; \theta^*), \hat{y _i})$, where $\hat{y _i} = y _i + 1$ and $\theta^*$ is the pre-trained model.
  • Both [1] and [2] aim to prevent randomly initialized models from learning useful semantic information.

• Test-time Data Poisoning:

  • TePA generates poisoned samples by maximizing entropy on the source model, aiming to blur the classification boundaries of the online model.
  • Our proposed method generates poisoned samples by constraining their feature distribution to closely match that of the samples before poisoning, causing the online model to learn incorrect information about the internal distribution.

  Both approaches aim to make the source model forget the source domain knowledge and distort the classification boundary.

Additionally, we will supplement experiments using methods from [1,2] to generate poisoned samples and inject them into the online model following the RTTDP protocol. The results for the CIFAR10-C dataset are shown below, and we will include results for CIFAR10/100 and ImageNet datasets in the camera-ready version.

[1] Unlearnable examples: Making personal data unexploitable

[2] Adversarial examples make strong poisons

W2: The Technical Contributions of Our Proposed Method.

We recognize that the main technical contribution of our method lies in the feature consistency regularization, which ensures that poisoned samples effectively attack the TTA model. We provide detailed derivations and explanations in the manuscript to support this.

Regarding attack objectives, we categorize existing poisoning methods into two types: high-entropy and low-entropy attacks (as introduced in the manuscript). We then analyze the limitations of these objectives and improve them by proposing the BLE Attack and NHE Attack, respectively.

W3: Related Work.

Thank you for your suggestion. We will move the related work section to Chapter 2 to provide readers with a preview of the TTA methodology. The specific TTA methods are described in the "Benchmark TTA Methods" subsection of the Appendix due to space limitations in the main body.

Q1: Does this only work against losses $\mathcal{L} _{tta}$ that are unsupervised?

Yes, $\mathcal{L} _{tta}$ is an unsupervised loss used in TTA models. For example, TENT minimizes the entropy of test samples, i.e., $\mathcal{L} _{tta} = E _{x _i \in \mathcal{B} _t} \sum_k -p _{ik} \log p _{ik}$, where $p _i = f(x _i; \theta _t)$. EATA minimizes entropy while also including a Fisher regularization term to prevent forgetting source domain knowledge, i.e., $\mathcal{L} _{tta}=E _{x _i\in\mathcal{B} _t}Entropy(f(x;\theta _t)) + \beta R(\theta _t, \theta _0)$.

Q2: Do you specify the attack budget?

Yes, in the "Hyperparameters" subsection in the Appendix, we specify that the attack budget $r$ is 50% throughout the experiment. We also perform an ablation study on $r$ in Table 8. Additionally, we have corrected the confusing notation in A.3.2, where $r = 0.1, 0.2, 0.5$ was mistakenly written as $b = 0.1, 0.2, 0.5$ in the original manuscript.

Q3(A): What are the "Source" numbers listed in Tables 2, 3, and 4?

The "Source" numbers in these tables indicate the performance of the source pre-trained models tested directly on the test stream, without using TTA updates. Typically, TTA improves the performance of the pre-trained source model on the test data stream. However, when poisoned data is injected into the test stream, unsupervised TTA methods may harm the model's performance. More details about the TTA methods can be found in the "Benchmark TTA Methods" subsection of the Appendix.

Dear Reviewer AP94,

Based on your valuable feedback, we have supplemented our experiments to compare with two data poisoning methods, i.e., Unlearnable Examples and Adversarial Poisoning, and more detailed clarifications on Comparison between our proposed method and existing works to highlight our challenges and Our technical contributions. We have also updated the revision as you suggested. As we approach the end of the discussion period, we would be grateful if you could review our response and consider our revised manuscript. Your constructive comments have helped us to significantly improve our work, and we believe that we have thoroughly addressed your concerns.

Thank you very much for your time and detailed review. We welcome any further additional questions or requests for clarification.

Yours sincerely,

The Authors

Dear Reviewer AP94,

With the discussion period ending in 10 hours, we would greatly appreciate hearing any further questions or concerns you may have. We are fully prepared to provide additional clarifications or responses as needed.

To summarize, in our rebuttal, we have:

• Conducted supplementary experiments comparing our method with two data poisoning approaches, Unlearnable Examples and Adversarial Poisoning.
• Provided detailed clarifications on the comparison between our proposed method and existing works, highlighting the challenges addressed and our technical contributions.
• Revised the manuscript in line with your valuable suggestions.

We are pleased to note that our constructive discussions and revisions have resonated positively with other reviewers. For example:

• Reviewer 8xpV was fully convinced by our additional analysis on the limitations of query methods, details of proposed assumptions, and evaluation protocols.
• Reviewer hmy9 expressed satisfaction with the clarifications regarding online vs. offline protocols, additional evaluations against strong adversarial attacks, and our analysis of various design choices.

We believe continued dialogue can further enhance the quality and clarity of this work. We look forward to your response and are happy to address any remaining questions.

Thank you once again for your thoughtful feedback and support!

Best regards,
The Authors

W1: Citation format error

Thank you for pointing this out. We have corrected the citation format in the revised manuscript.

W2: Clarification for Figure 1

We appreciate your valuable suggestions. To address the confusion, we will expand the descriptions of the notations in the caption of Figure 1. Specifically:

• $\mathcal{B} _{ab}$ and $\mathcal{B} _a$ denote the adversary's samples, where $\mathcal{B} _{ab}$ represents the benign samples prior to poisoning, and $\mathcal{B} _a$ refers to the poisoned samples.
• $\mathcal{B} _b$ represents normal/benign user samples.
• $\mathcal{B} _t$ denotes the input data to the TTA pipeline at timestamp $t$.

As suggested, we will further refine Figure 1 in the revised manuscript for improved clarity.

W3: Graphical classification of threat models

Thank you for the suggestion. We will include a graph classifying the threat models used in various attack methods. This addition will be provided in the Appendix of the revised manuscript.

Q1: Impact of the ratio between benign and poisoned samples

We have analyzed the impact of varying poisoned sample ratios in Table 8 of the Appendix. The results demonstrate a reasonable trend: the error rate of benign samples predicted by the online model improves gradually as the proportion of poisoned samples increases.

Q2: Comparison with harder attacks requiring access to benign examples

Thank you for recommending valuable related work[1]. We will incorporate a discussion of this method in our revised manuscript. Specifically, we have implemented the GMSA attacks[1], including GMSA-MIN and GMSA-AVG, in the context of our RTTDP protocol. In this setup, poisoned samples were generated using the GMSA attack methods and subsequently injected into the TTA model. The attack success rate was then evaluated on other benign user samples. The results on CIFAR10-C dataset with Uniform attack frequency are shown in the below table.

[1] Towards evaluating the robustness of neural networks learned by transduction.

Q2&3: Is it possible to derive any formal guarantee on the attack effectiveness when relaxing the different constraints?

In our ablation study, as presented in Table 5, we analyzed scenarios involving access to online model weights and the impact of excluding the feature consistency constraint. Our observations are as follows:

1. Impact of Feature Consistency Constraint

   • In the absence of a feature consistency constraint, low-entropy attacks typically generate poisoned samples that fail to effectively influence benign samples through TTA updates.
   • Conversely, high-entropy attacks, such as those that maximize entropy or utilize NHE, prove to be more effective. This is because the TTA gradients computed from poisoned samples with high-entropy predictions are significantly larger, causing substantial perturbations to the source model during updates.

2. Effectiveness of Feature Consistency Constraint

   The inclusion of our proposed feature consistency constraint greatly enhances the effectiveness of the attack, regardless of whether poisoned samples are generated using the surrogate model or the online model.

3. Access to Online Model Weights

   The average attack success rate across all TTA methods is highest when the attacker has access to the online model, enabling the generation of more accurate poisoned samples.

On the other hand, assuming access to benign user samples presents its own challenges. If such access were feasible, an adversary could modify benign samples directly through adversarial attacks before they are injected into the online model, rather than introducing poisoned samples.

---

Thank you for your valuable comments. We hope the above clarifications and additions comprehensively address your concerns. Your insights have been instrumental in improving the quality of our manuscript. We sincerely appreciate your support and constructive suggestions!

Best regards,

The Authors

W1: Improved Assumptions Still Strong.

Thank you for your insightful comments. In this work, we relax two critical assumptions commonly made in prior research: (i) the poisoned samples are generated using a white-box (online) model, (ii) the poisoned samples are created by maximizing the error rate of benign users' (validation) samples. These assumptions represent significant limitations to the practical applicability of earlier studies.

In response to the points raised by the reviewer:

• We believe that obtaining the source model is relatively straightforward, as it is often a well-known model, such as an open-source foundation model or a pre-trained model derived from large-scale datasets. Additionally, an approximate source model can be distilled using a set of test data, further reducing the dependence on this assumption.

• It is more practical to obtain the distribution of benign users' data than to access their specific samples. For instance, an adversary could target data from a specific environmental condition (e.g., rainy or foggy settings) where the data distribution can be reasonably approximated.

We acknowledge the importance of exploring even more relaxed assumptions to enhance the practicality of our approach, and this will be a priority for future work. Nonetheless, we believe that our current study represents a significant advancement over existing methods, both in practicality and theory.

W2: Unclear Details.

We appreciate the reviewers’ careful reading of our manuscript and their valuable questions. Below are our detailed responses to each query:

Q1: L181: Why is repeated querying prohibited?

In traditional black-box attacks, adversarial gradients are estimated via repeated querying. However, obtaining a batch of poisoned samples through such methods typically requires thousands of queries. This approach is impractical in real-world online TTA scenarios because (i) excessive querying is easily detectable using straightforward monitoring strategies, and (ii) the TTA model updates with each query, rendering gradient estimates unreliable. To clarify clearly, we will revise the term "prohibited" to "unavailable" in the manuscript.

Q2: How exactly is the model distilled?

As shown in Fig. 2(a), the distilled surrogate model suffices for generating poisoned samples, achieving performance comparable to the online model. In our method, the surrogate model is updated iteratively (10 iterations using Eq. 1) based on the feedback from the last batch of injected poisoned samples. Importantly, these updates use the fixed feedback, eliminating the need for repeated queries to the online model.

Q3: How can we assume to know $\mathcal{B}_{a,t-1}$?

The notation $\mathcal{B}_{a,t-1}$ refers to poisoned data injected in the previous round. Between two launching attacks, an unknown number of benign user samples enter the TTA model. The surrogate model need not align perfectly with the real-time model parameters $\theta_t$; instead, it is sufficient for the surrogate model to approximate the lagged parameters $\hat{\theta} _t \approx \theta _{t-\delta}$, where $\delta$ indicates the timestamp gap between two batches of injected poisoned data. We will clarify this in the revised manuscript.

Q4: What do Uniform and Non-Uniform Attack Frequencies refer to?

These terms are defined in the “Evaluation Protocol” subsection of the Appendix due to space limitations in the main text. Uniform refers to poisoned samples being uniformly injected throughout the TTA pipeline, while Non-Uniform refers to concentrated injections at the beginning of each test domain.

Q5: What does "Source" refer to?

"Source" in Tables 2–4 denotes the baseline inference performance of the source pre-trained model without test-time adaptation. Details about TTA methods are provided in the “Benchmark TTA Methods” subsection of the Appendix.

Q6: The meaning of L259–L260.

Lines 248-257 discuss the inefficiency of generating poisoned samples using Eq. 3, as $\mathcal{B} _a$ only harms the TTA model when jointly injected with $\mathcal{B} _{ab}$, but it would waste half of query budget. The mismatch in feature distributions between $\mathcal{B} _a$ and $\mathcal{B} _{ab}$ causes normalization statistics in batch normalization (BN) layers to differ when $\mathcal{B} _a$ is forwarded alone versus with $\mathcal{B} _{ab}$. To address this, in lines 259-260, we introduce an additional constraint, $D(P _a, P _{ab}) = 0$, combining the optimizating objective $\mathcal{B}_a$ and optimized objective $\mathcal{B} _{ab}$ into a single objective. We will clarify this notation in the revised manuscript.

Q7: The meaning of L431.

Line 431 states that our proposed surrogate model outperforms the source model in generating poisoned data for PGD attacks and even slightly exceeds the online TTA model in Table 5. We will rephrase this sentence for clarity.

Q8: The meaning of L149/150.

In realistic scenarios, the attacker is unable to access the real-time model weights. The assumed used into this sentence is meant to correspond to the reason "The update is carried out on the cloud side" stated before. We will clarify clearly in the revised manuscript.

Q9: What does "Offline" attack order refer to?

The TePA protocol is classified as “Offline” because all poisoned samples are injected into the TTA model before any benign user samples are processed. In contrast, our RTTDP protocol injects poisoned samples into the TTA model alongside benign samples throughout the test stream, enabling online updates using mixed data. Thus, RTTDP is classified as an “Online” attack.

We hope these responses address the reviewers' questions comprehensively.

W3: No Code Available.

We promise to release our all code, including the implementations of our proposed method and all competing methods, to provide more implemented details as soon as this manuscript is accepted.

For the improvement suggestions.

We thank the reviewer for the efforts to help us improve our manuscript representation. We would update the Figure 1 and Tables 5-7 according to the suggestions in the revised version. Due to the limited space, we could not break Figure 2 into individual figures.

---

Thank you for your valuable comments. We hope the above clarifications and additions comprehensively address your concerns. Your insights have been instrumental in improving the quality of our manuscript. We sincerely appreciate your support and constructive suggestions!

Best regards,

The Authors