Scanning Trojaned Models Using Out-of-Distribution Samples

Thank you for the valuable comments. Responses to specific points are provided below:

W1:

• We apologize for the referencing error. We intended to refer to Figure5 in SectionE.

• We mentioned that near-OODs are those that share semanti/stylistic features with the IDs making them harder to distinguish (e.g., CIFAR10 vs. CIFAR100). On the other hand, far-OODs do not share any semantic or stylistic similarity with IDs (e.g., CIFAR10 vs. MNIST), making them easier to identify as OODs. For a more formal definition this Concepts, we kindly refer the reviewer to [1], as this is not the primary focus of our study; instead, we leverage these concepts for detecting trojaned classifiers.

• The main motivation behind TRODO is to define and find a general and robust signature that distinguishes trojaned and clean classifiers, independent of architecture, label mapping strategy in trojaning, and the training strategy of the classifier (clean/adversarial).

---

W2:

As stated in the caption of Table 1, all ACC* columns are related to adversarially trained models. Additionally, lines 287-290 mention, "Specifically, TRODO achieves superior performance with an 11.4% improvement in scenarios where trojan classifiers have been trained in a standard (non-adversarial) setting and a 24.8% improvement in scenarios where trojan classifiers have been adversarially trained," indicating a greater improvement on adversarially trained models. Moreover, Table 2 in our paper includes columns for rounds 3, 4, and 11 of TrojAI, which contain adversarially trained models.

Table 1: Performance of TRODO compared with other methods, in terms of Accuracy on standard trained evaluation sets (ACC%) and adversarially trained ones (ACC*%).

---

W3:

We aimed to provide similar ablations in Table 3 of the paper. However, to address your concerns, we conducted additional experiments.

In the first experiment, we performed an ablation study on the number of samples in our validation set. By default, TRODO Zero uses the Tiny ImageNet validation dataset, which contains 200 classes, each with 500 samples. We explored the effect of varying the number of samples per class and investigated the ratio [0.0, 1.0].

Here, we utilized the DC metric [2] to measure the diversity based on your suggestion. A higher value indicates greater density and coverage of the validation set with respect to the evaluation set:

---

Q1:

We apologize for the missing details about hard augmentations/transformations and kindly ask you to review our common response, where we have provided detailed answers to this question.

---

Q2:

Here for each validation set, we crafted the OOD samples with hard transformations and then computed their distance using the FID metric:

---

Q3:

We apologize for any ambiguity/inconvenience caused by using different terms. However, we would like to clarify that our method is a post-training defense approach aimed at detecting whether an input model/classifier has been trojaned (backdoored). After reviewing the literature, we understand that researchers refer to this as both 'trojan detection methods' and 'scanning methods,' as the task involves scanning an input model.

[1] Winkens Contrastive 2021

[2] Naeem Reliable 2020

Thank you for your useful comments. Please find our responses below:

W0.1&W0.2:

We appreciate your suggestions and will implement them to improve clarity and logical flow in the final manuscript based on your recommendations.

---

W1:

We conducted additional experiments to establish an adaptive Trojan attack. We have detailed these in our common response, which we kindly ask you to review.

---

W2:

As stated in limitation section of our paper, for each new architecture, we have to firstly tune $\epsilon$ and then $\tau$. For each selection of architecture and validation dataset, first we have to train a surrogate classifier $g$ on the selected validation set and then find $\epsilon$ using DeepFool [1]. As the final stage for computing $\tau$ is not time consuming, we only report the computational cost of the first two stages of hyperparameters tuning in the tables below (the values are in terms of hours):

• The experiments have been conducted on a RTX 3090.

Regarding the sensitivity of TRODO to the validation set and hyperparameters, we have provided the values of hyperparameters for various selections of architecture and validation dataset:

ResNet18

PreActResNet18

ViT-B/16

Additionally, the performance results of our method using these hyperparameters can be found in Table 3 in the Ablation Study (Section 6) of the paper. This section demonstrates the robustness and effectiveness of our method across different datasets in the validation set.

---

W3:

We apologize for the oversight and any confusion it may have caused. It seems there was an issue that resulted in the exclusion of results for PreActResNet18 and ViT-B/16 models from Appendix Section M. We will ensure that these results are included in the final updated version of the paper. Below are the detection results in terms of accuracy on standard trained evaluation sets (ACC %) and adversarially trained ones (ACC* %).

PreAct ResNet-18 Architecture:

ViT-B-16 Architecture:

---

Q1:

We apologize for the missing details about hard augmentations/transformations and kindly ask you to review our common response, where we have provided detailed answers to this question.

---

Q2:

We appreciate your insightful question. Our primary objective is to develop a general method with a consistent pipeline regardless of the training data distribution. We aimed to avoid making dataset-specific modifications, such as filtering out overlapping classes for CIFAR-10 but not for others, such as GTSRB. By not removing common classes, we ensure that our approach remains uniform and applicable to various datasets without additional preprocessing steps.

---

L1&L2:

Although our method can operate without access to any training data, we confirm that having access to such data from the input classifier can improve our detection performance. However, it should be noted that all existing detection methods, except for MM-BD, strongly rely on training data for operation. On the other hand, our method works in scenarios where training data is unavailable by just leveraging a small validation set. As shown in Table 3, by using only the FMNIST or SVHN dataset, we achieve superior performance, outperforming other methods by up to 10%.

[1] Moosavi-Dezfooli et al. DeepFool 2015

Thank you for your thoughtful feedback and for taking the time to review our rebuttal! We greatly appreciate your careful evaluation and positive response.

Sincerely, The Authors

Thank you for your thoughtful review. We have provided the following response:

W1.1:

We believe the epsilon value of the attack plays a key role compare to steps in our setup. If epsilon is large, as you mentioned, our signature for both clean and Trojan classifiers would be the same. This is the main reason we carefully estimate it using the validation set and Boundary Confidence Level to avoid such scenarios (see line 199). The number of steps in the attack has less effect as the attack radius has already been determined (Madry et al. Toward 2017). To further address your concern, we have provided an additional experiment where every component of our pipeline remains fixed except for the number of steps of the attack:

---

W1.2:

It is possible that temperature rescaling affects the regular softmax function, but the classifier remains more confident in in-distribution samples compared to OOD samples (Tajwar et al., No True 2021). This is consistent with the rationale and principle behind our method. As a result TRODO, would also detect trojaned classifiers with temperature scaling. To further clarify this concern, we kindly ask you to refer to our common response, where we considered worst-case scenarios involving adaptive attackers. Even when the attacker is aware of our defense mechanism, TOROD demonstrated robust and consistent performance.

---

W1.3:

The mentioned papers, CCAT (Stutz et al., 2020) [A] and RATIO (Augustin et al., 2020) [60], aimed to enhance robust performance on in-distribution samples. Improving robustness on adversarially out-of-distribution (OOD) samples was not their main purpose. Although they considered the issue of overconfidence on OOD samples, their primary goal was robust in-distribution classification. They do not weaken principle behind TRODO, which is based on the vulnerability of existing robust methods, including CCAT, RATIO, and other SOTA methods, to perturbed OOD samples, especially when they are close to the in-distribution, referred to as near OOD in our paper (see line 88 and Table 5). This observation has also been demonstrated by Fort (Adversarial, 2022). Moreover, recently, RODEO (ICML, 2024) has shown that in the CIFAR10 vs. CIFAR100 OOD detection challenge, there is no better performance than random detection (i.e. less than 50% AUROC), which further supports our claim (see their Table 4-a of our paper). Additionally, we kindly ask you to check our common response, where we show that even in the worst-case scenario (adaptive attack), TRODO demonstrates promising performance.

Finally, to further address your concern, we trained input trojan models using the method proposed by RATIO instead of common adversarial training and reported TRODO's performance on them here. Other components of TRODO, including the training pipeline, remain fixed.

---

W2:

Intuitively, increasing the poisoning rate enlarges the blind spots in trojaned classifiers, as these are boundary regions where the poisoned data causes the model to overfit. Consequently, this will increase the probability that TRODO detects the trojaned classifiers. However, our signature is based on the presence of blind spots in trojaned classifiers and shows consistent performance across different poisoning rates. In the paper, we considered a poisoning rate of 10% as it is common in the literature. In response to your comment, we have provided TRODO's performance for different rates below: (Other components of TRODO remained fixed.)

---

Q1:

Thank you for your insightful question. You are correct that the score $S_i$ can take values only within a specific range, as confidence is upper bounded by $1$. We apologize for the oversight in the paper where we did not mention that we apply a transformation to the score $S_i$. In our method, we use $-log(1 - S_i)$ instead of $S_i$ directly. This transformation maps the original score to a new range $[0, ∞)$, making it suitable for fitting with a left-truncated normal distribution. We will include this clarification in the final manuscript to ensure there is no confusion.

We thank the reviewer for their thoughtful review and valuable feedback.

• A key requirement for deploying deep neural networks in real-world applications is calibrated or approximately calibrated behavior on input samples. A miscalibrated classifier, particularly in cases where $T→0$, $T>>1$, can lead to poor decision-making. For example, in medical diagnosis, overestimating disease probability can result in unnecessary treatments, while underestimating it may lead to missed diagnoses. Miscalibrated models are especially unreliable in high-stakes scenarios such as finance, healthcare, or autonomous systems, where accurate probability estimates are crucial. Furthermore, the backdoor attack/defense literature generally assumes that Trojaned classifiers behave normally on samples without triggers, similarly to standard classifiers. Since standard classifiers typically exhibit approximately calibrated outputs, we implicitly assume that the Trojaned classifier would also retain this characteristic.

• We acknowledge the challenges posed by the extreme temperature scenarios, specifically when $T→0$ or $T>>1$, as our ID-score would converge to zero in both cases. However, we believe this issue can be addressed with minimal modifications. In the backdoor attack setup (as detailed in our threat model), it is commonly assumed that the attacker has access to the input model. To mitigate the issue, we propose a minor extension to TRODO: applying its own softmax function to the logits of the input classifier instead of relying on the input classifier's final softmax output. This approach could help counter the adversarial temperature settings highlighted by the reviewer.

• Moreover, since TRODO and TRODO-Zero both utilize an OOD set, another minor extension could involve evaluating the classifier's softmax output on these samples. If the output significantly deviates from a uniform distribution (e.g. measured by Kullback–Leibler divergence), the model could be rejected as extremely miscalibrated due to the attacker's manipulation.

To further address your concern, we evaluated TRODO's performance on an all-to-one label mapping task using the CIFAR-10 and GTSRB datasets across different temperature values, while keeping other components unchanged. The results show that our designed detector effectively handles a reasonable range of temperature values, maintaining consistent performance throughout.

Am I missing something?

• We believe there has been some misunderstanding. When we mentioned that 'the classifier remains more confident in in-distribution samples compared to OOD samples,' we were referring to Fact-A. We would like to clarify that the principle behind our signature remains valid even when temperature rescaling is applied. The logical flow of our principle is as follows: Fact-A ⇒ Fact-B ⇒ Fact-C ⇒ Fact-D ⇒ Fact-E ⇒ Fact-F ⇒ Fact-G.

• Furthermore, as indicated in Fact-A, the softmax output tends to resemble a uniform distribution for OOD samples. As a result, applying temperature rescaling will affect all logits equally and will not alter this uniform shape. Consequently the ID score for clean OOD samples would still be very low (e.g., approximately $\frac{1}{N}$, where $N$ is the number of ID classes). However, by perturbing these samples, the ID score can increase (up to one). Therefore, our signature would not be arbitrarily small as mentioned; it remains within the range of ([0,$1-\frac{1}{N}$]).

Fact-A: A classifier remains more confident in IDs compared to OODs. Specifically, the output softmax of a classifier for OODs tends to be more uniform, while for IDs, it is more concentrated. (confidence: maximum softmax probablity) [1].

Fact-B: Previous research has primarily focused on perturbing IDs, shifting them, and using this as a signature to distinguish between clean and trojaned classifiers [2,3,4,5,6,7,8,9,10].

Fact-C: In scenarios where classifiers have been adversarially trained, perturbing ID samples for signature extraction becomes less effective. This is because adversarial training enhances the classifier's robustness to such perturbations, reducing their impact [9,10].

Fact-D: Instead of perturbing IDs, perturbing OODs toward the in-distribution is a viable approach, as classifiers are vulnerable to this shift. This approach remains effective even for adversarially trained classifiers, as they are still susceptible to perturbed OODs. This finding is also supported by parallel research in OOD detection [11,12,13,14].

Fact-E: Trojaned classifiers learn decision boundaries that include 'blind spots,' which are intuitively regions along the in-distribution boundary that have been altered to overfit on poisoned training data (data with triggers), thereby changing the geometry of the boundary.

Fact-F: Perturbing OODs toward the classifier's decision boundary increases their ID scores, and this effect is particularly significant in trojaned classifiers. This is because the perturbations can exploit the blind spots within the decision boundary, mimicking the triggers used during the trojan attack.

Fact-G*: We use the difference in ID scores between a benign OOD sample (without perturbation) and an adversarially perturbed OOD sample as a signature to differentiate between clean and trojaned classifiers.

---

What was the OOD...?

Following their reported setting, we used Tiny ImageNet as the out-of-distribution dataset for training.

---

In general, these methods.. .

Regardless of the adaptation strategy, existing classifiers are vulnerable to adversarial attacks on OODs, particularly when these samples are close to the in-distribution boundary. This vulnerability, as demonstrated recently by (Lorenz Deciphering, 2024), (Fort Adversarial Vulnerability, 2022), and others [11,12,13,14], limits them to act as counters to our method. To further illustrate this, we conducted additional experiments using RATIO as an OOD detector, where the MSP is employed as the ID score. In the first scenario, CIFAR-10 serves as the ID dataset and CIFAR-100 as the OOD, and vice versa in the second. While the method achieves good performance in clean scenarios , in adversarial scenarios—where perturbations are added to shift OODs to in-distribution and vice versa—its performance drops to below random detection, highlighting its vulnerability to perturbed OODs.

---

I think the additional ...

In the adaptive attack scenario, we evaluated various label mappings and adaptive strategies. The worst performance decrease we observed was 12% (from 86.0% to 74.1%). Despite this, our method still outperforms previous detection methods that were not subjected to adaptive attacks. For instance, UMD achieves 57.7%, while our approach achieves 74.1%.

[1] Hendrycks, A baseline, 2016

[2] Xiang, UMD: Unsupervised , 2023

[3] Wang, Mm-bd: , 2024

[4] Wang, Neural, 2019

[5] Liu, Abs:Scanning, 2019

[6] Shen, Backdoor scanning for, 2021

[7] Guo, Tabor, 2019

[8] Hu, Trigger hunting, 2022

[9] Edraki, Odyssey, 2021

[10]Zhang, Cassandra, 2021

[11] Chen Robust OOD 2020

[12] Azizmalayeri, Your Detector 2022

[13] Chen, Atom ,2021

[14] Mirzaei, RODEO, 2024

Thank you very much for your positive feedback on our paper. We greatly appreciate your insights and suggestions.

W1:

We acknowledge that extracting hyperparameters may be considered a limitation of our method, as previously discussed in our limitations section. However, it is important to note that these hyperparameters must be selected based on the architecture where the validation is fixed. Moreover, our method demonstrates robustness with respect to the validation set, as evidenced in Table 3 of the Ablation Study. Specifically, using small datasets like FMNIST or SVHN for validation, our method outperforms others by up to 10%.

---

Q1:

We sincerely apologize for any confusion caused. To clarify, the training dataset is initially compromised using a backdoor attack. The classifier is then adversarially trained using the PGD (Projected Gradient Descent) attack method. As mentioned in the caption of Table 1, all columns labeled ACC* pertain to adversarially trained models. Additionally, lines 287-290 states: "Specifically, TRODO achieves superior performance with an 11.4% improvement in scenarios where trojan classifiers have been trained in a standard (non-adversarial) setting and a 24.8% improvement in scenarios where trojan classifiers have been adversarially trained," indicating a significant improvement in adversarially trained models. Furthermore, Table 2 includes columns for rounds 3, 4, and 11 of TrojAI, which involve adversarially trained models.

Table 1: Scanning performance of TRODO compared with other methods, in terms of Accuracy on standard trained evaluation sets (ACC %) and adversarially trained ones (ACC* %).

Table 2: Comparison of TRODO and other methods on all released rounds of TrojAI benchmark on image classification task. For each method, we reported scanning Accuracy and the average time scanning time for a given classifier.

---

Q2:

We thank the reviewer for the suggestion. We should note that access to a portion of the training data would enable us to create near-OOD samples with higher quality compared to similar methods. Specifically, the mentioned method's strategy for exploiting data from a classifier leads to samples with artifacts and shortcuts, reducing its effectiveness as it becomes more apparent to the model that they do not belong to ID (considering them far OOD). However, to further explore your suggestion, we evaluated both the proposed method and the FastDFKD (Fang Data-free 2022) method for crafting data in situations where there is no access to training data:

---

Q3:

Thank you for pointing out the formatting inconsistency. We apologize for this oversight and will ensure that all paragraph headings are uniformly formatted in the final manuscript.

---

L1:

We acknowledge that the limitation you mentioned is valid. However, the scenario of having no clean training data is a special case. Most research studies, including ours, typically utilize at least a small amount of clean data to validate their methods.

We appreciate your insightful review. Here is our detailed response:

W1:

We sincerely apologize for this oversight. There were some issues with the command that removed these tables from our submitted paper. Here, we have provided those tables and assure you that they will be included in the final manuscript. We should note that in the following experiments, all components of TRODO are fixed except for the architectures of the classifiers.

Below are the detection results in terms of accuracy on standard trained evaluation sets (ACC %) and adversarially trained ones (ACC* %). Due to character limits, we have included only the top competitors from the methods considered in Table 1 of our paper.

PreAct ResNet-18 Architecture:

ViT-B-16 Architecture:

---

W2: Thank you for suggesting this evaluation, and we apologize for missing it in our submitted version. To address the mentioned concern, we have provided different scenarios of adaptive attacks in our common response, which we kindly ask you to review.

---

W3:

We acknowledge that the paper lacks an intuitive explanation connecting the theoretical analysis to the proposed method. The theory section aims to provide high-level intuition for the method's core principle: trojaned models are more susceptible to adversarial perturbations, especially in near-OOD regions.

As deducible from our experiments, there is a clear performance gap between TRODO and TRODO-Zero, and the main reason for this gap is the utilization of near-OOD samples instead of arbitrary (mostly far-OOD) ones. We have also illustrated this phenomenon on Figure 2 of the paper. Using near-OOD samples enhances the change of ID-Score, resulting in more recognizable signature for trojan scanning. To further validate this intuition, we have provided Theorem 1, in which we prove that the adversarial risk on near-OOD data is more evident.

In Theorem 2, we analyzed a simplified scenario using a least-square loss and two-layer networks. Despite the simplicity, these cases are indicative of the general scenario because other complex losses used in practice yield similar optimization outcomes. Additionally, two-layer networks are known to be universal approximators capable of learning any function, analogous to more complex architectures like ResNet with MSE loss. This theoretical foundation helps to explain the observed phenomena in more intricate setups, thereby reinforcing the validity of our proposed method. Regarding the connection of this Theorem with our work, it is noteworthy that the core principle of TRODO is to use the difference of ID-Score of OOD samples before and after attack as the scanning signature. This change of ID-Score is equivalent to the adversarial risk which we have defined in Section 4. According to this theorem, this risk is linearly bounded by the trigger norm ($||t||$), which is non-zero for backdoored models and 0 for clean ones, making them distinguishable by our signature.

In future revisions, we will elaborate on this connection to provide readers with a clearer understanding of how our theoretical analysis supports and motivates the proposed method.

Thank you for your valuable comments. Our responses to each point are provided below:

Q1&Undefined Threat Model

Sorry for the confusion. We have briefly stated our threat model in lines 212-226 of the paper. We further present our threat model more clear in depth here. We assure the reviewer that we will improve the clarity of this section in our final manuscript.

Attacker Capabilities

• Data Poisoning and Training Influence: Attackers can poison training data [1, 3] and influence the training process [2, 4] to embed backdoors into models.
• Trigger Visibility and Coverage:
  • Stealthy to Overt Modifications: Attackers can deploy triggers that range from undetectable to more noticeable modifications.
  • Local and Global Coverage: Triggers can affect specific parts of a sample [1, 4] or the entire sample [5, 6].
  • Sample-Specific Attacks: Attacks can be tailored to specific samples [7], complicating detection.
• Label-Consistent Mechanisms: Attackers can use poisoned inputs labeled according to their visible content, which leads to misclassification during inference [8, 5].
• Attack Types: Attacks can be either All-to-One or All-to-All. In the former, a single target class is selected and whenever the input contains the trigger, the classifier outputs the selected target class. In the latter however, there is more control over the attack. For each source class (the class to which the clean input actually belongs), an arbitrary target class can be chosen so that the presence of the trigger causes the model to classify the input as that target class.
• Model Training: Models can be trained adversarially or non-adversarially.

Attacker Goals

• Embed Backdoors: Ensure the model contains backdoors that lead to misclassification during inference.
• Maintain Stealthiness: Create triggers that may be undetectable or hard to detect, even under scrutiny.
• Evade Detection: Implement attacks that complicate detection efforts, especially those that are sample-specific or label-consistent.

Defender Capabilities

• Model-Only Detection: The defender receives the model and may (TRODO) or may not (TRODO-Zero) have access to a small set of clean samples from the same distribution as the training data.
• No Prior Knowledge Required: The detection mechanism operates without any prior knowledge of the specific type of attack or the nature of the trigger involved.

Defender Goals

• Detect Backdoors: Identify if the model has been compromised with a backdoor.
• Adapt to Various Scenarios: Effectively scan and detect backdoors both in scenarios with and without clean training samples.

---

Q2&Lack of Comparison with SOTA Baselines

We have compared TRODO with eight strong Trojan detector methods in our paper. To further address your concerns regarding the comparison with other baselines, we have conducted additional experiments, including comparisons with FreeEagle, STRIP, ANP, Ex-Ray, and DF-TND. The results of these comparisons are presented in the following table:

Although FreeEagle aims to be effective against various types of trojan attacks and performs better than other baselines in this table, it is particularly vulnerable in All-to-All settings, where samples from each class can be mapped to different target classes in the presence of a trigger. FreeEagle primarily addresses the scenario where a single source class is mapped to a single target class and attempts to identify such pairs, if they exist. Thus, it can only perform well in All-to-One scenarios.

---

The results listed in Table 2 (baseline-NC) differ significantly ...

Regarding the results of NC on the TrojAI benchmark (Table 2 in our paper), it is important to note that our results are specific to the TrojAI benchmark, which is not covered in FreeEagle’s Table 4. We have included baseline results from K-arm's [9] Table 1, which is a baseline primarily focusing on this benchmark. We will ensure these details and the additional comparative analysis are explicitly included in the revised version to enhance clarity and comprehensibility.

[1] Gu et al. Badnets 2017

[2] Nguyen et al. Wanet 2021

[3] Chen et al. Targeted 2017

[4] Nguyen et al. Input-aware 2020

[5] Barni et al. New backdoor 2019

[6] Wang et al. Bppattack 2022

[7] Li et al. Invisible 2021

[8] Turner et al. Label-consistent 2019

[9] Shen et al. Backdoor scanning 2021

Thank you for your positive feedback and for considering a higher score for our work! We will ensure that these experiments are incorporated to further enhance the manuscript.

Sincerely, The Authors