W1&Q1. Unclear threat model: The paper claims a semi-honest adversary, but the adversary controls training and may embed backdoors — this is not honest-but-curious.

A1. Thanks. We misused the term "semi-honest adversary": while the notion of a potentially manipulated training process is widely adopted in the backdoor detection literature, the term "semi-honest" has a strict definition in cryptography (i.e., honest but curious), which we overlooked.

In fact, the threat model we adopt is more accurately described by the machine learning community’s notion of an “untrusted third-party provider”: the model is trained externally by a service provider who may deviate from the intended protocol and insert backdoors, while detection is performed independently by the user (a trusted entity). The attacker (untrusted provider) has full control over training but no knowledge of the detection method, while the defender (user) has no access to training data or process—consistent with prior works such as DECREE.

Accordingly, we will revise the manuscript as follows: Replace all mentions of "semi-honest adversary" with "untrusted third-party provider" to avoid ambiguity.

Modify the beginning of Section 3.1 (“Threat Model”) to:

We consider a practical outsourced training scenario, where model training is performed by a third-party provider who may nominally follow the protocol but is essentially untrusted and may insert backdoors. The user has no visibility into the training process, but can independently inspect the returned model for malicious behavior.

W2&Q2&Q3. Strong assumptions. (a) The paper assumes a non-adaptive attacker unaware of the detection, unlike prior works with adaptive settings, weakening robustness claims. (b) Collusion between providers injecting identical backdoors is realistic but not evaluated.

A2 (a). We follow the adaptive attacks similar to TED and MM-BD, which simulates adaptive attackers by modifying the training process of the backdoor model.

We design an adaptive attacker that is aware of our detection and aims to bypass our CKA-based representation divergence detector. In particular, the attacker first trains a clean reference model $f_{clean}$ on the clean training dataset. Then, when training the adaptive backdoored model $f$ on the poisoned training dataset, the attacker combines the original training loss $L_{origin}$ with a regularization term designed to suppress detection. This regularizer maximizes the CKA similarity between the clean and backdoored models:

$$L_{adaptive} =L_{origin} - \lambda \cdot L_{CKA}(f, f_{clean})$$

$L_{CKA}$ denotes the CKA similarity between intermediate representations of the two models under the same input as used in our paper, and $\lambda$ controls the strength. We vary the $\lambda$ and report the attacker’s ASR as well as our defense’s DSR and FPR in the table below.

From the results, we conclude:

• Adaptive attacks do reduce detection accuracy: As $\lambda$ increases, the attacker can partially suppress cross-model representation divergence, causing our DSR to degrade from 99.61% to 86.54%.
• The attacker must sacrifice backdoor effectiveness: The ASR drops from 98.75% to 82.37%, indicating that the backdoor behavior itself is weakened. This reveals a trade-off between stealth and efficacy，as attackers cannot simultaneously achieve high attack success and high stealth.

In summary, our method remains robust even under defense-aware, adaptive threats, raising the barrier and cost for stealthy backdoor injection.

A2 (b). We believe there may be a misunderstanding. In our experiments, we randomly sample model pairs from a pool containing both clean and backdoored models. Thus, all three combinations are covered: clean-clean, backdoor-backdoor, or clean-backdoor. Importantly, backdoored models share the same injected trigger. The strong performance reported in our paper reflects our method's robustness under all these possible pairings.

1. In Appendix E, we consider the strongest model collusion scenario, where two service providers share the same poisoned data, model initialization, and training strategy to implant identical backdoors. In our response A1 to Reviewer VC1P, we proposed a more systematic evaluation for collusion-based backdoor attacks, introducing four levels ranging from L0 (completely independent) to L3 (strong collusion). We conducted experiments on three attacks and three tasks. Results show that our method maintains performance superior to existing SOTA methods even under weak collusion (L1), such as achieving an 85% DSR on TrojanVLM compared to the existing method's 80%.
2. Collusion is permitted in extreme cases, to reduce the risk of detection failure, we randomly assign training data to independent parties, ensuring their backdoor samples do not overlap from the source.

W3&Q4. Clean data dependency unclear: $\mathcal{D}_s$ is sampled from $\mathcal{D}_c$, which may imply access to clean data.

A3. In the original manuscript, we stated: “The verifier user (the police) has no access to the training data or process and cannot assume the availability of a clean reference model.” We acknowledge that this may cause confusion. To clarify: in our setting, the verifier cannot access the full training dataset used by the providers, nor can they observe the actual training process. However, the verifier may possess a small subset of clean data, either from their own held-out inputs or collected from the same domain. This subset is used for trigger optimization and detection. We revise the statement for clarity as:

“The verifier cannot access the full training data or training process on the provider side but may have access to a small, trusted subset of clean data. No clean reference model is assumed.”

This accurately reflects our assumption: detection relies on a small amount of verifier-held clean data, not on any privileged access to the full training set.

W4. Evaluation unfair: majority vote assumes multiple models, deviating from the original two-provider setting.

A4. Thanks. Below we clarify the intent and fairness of our evaluation design:

In “Configuration”, “We randomly select $N(N-1)/2$ pairs of models from $N$ models and apply voting.” What we actually meant was: From the $C(N, 2)$ possible pairs of models formed by the $N$ models, we randomly select 20 model pairs (without repetition) for testing, and compute the detection performance by averaging scores. We use “voting” in the original text was inaccurate. In the initial draft, the term “voting” was primarily used to summarize the concept of “enhancing robustness through multiple comparisons,” but it failed to accurately distinguish between soft aggregation and hard majority voting. Thanks for pointing out this error and have unified the terminology in the final version to “average of scores from multiple comparisons.” Additionally, to further address the concern about the fairness of the evaluation, we include results using all $C(40, 2) = 780$ model pairs, and then average the results.

ResNet-18

CLIP

LLaVA

Compared with the results in the main text, the average results did not show a significant decline in performance. The detection success rate and false positive rate remained stable, with an average variance of less than 0.5, and *consistently outperformed all baseline methods across all tasks and model types.

W5. Generalizability — OD Loss seems to require clean labels, which may not be available in self-supervised settings.

A5. We clarify that the OD Loss in our framework is not tied to class labels or external references, but is instead defined as the negative of the original task loss under each learning paradigm (Eq. (5)).

In backdoor attacks against self-supervised learning (SSL) scenarios such as MoCo v2, BYOL, or MSF, where no labels are available [1], we use the negative of the SSL training loss (e.g., contrastive objectives). These losses are inherently label-free, making OD Loss applicable in such settings.

To validate this, we do experiments on ImageNet-100, which randomly selects 100 classes from ImageNet-1K and all training configurations follow the official setups in [1]. We support 10 clean and 10 backdoor samples to compute the average results.

Below are the detection results using our method under SSL backdoor attacks [1]:

Results show that our method generalizes well beyond supervised learning and performs robustly even in self-supervised settings without labels. Since OD Loss always operates on the model’s internal training objective, it remains applicable to any learning setting where a task loss is defined, without requiring access to labels.

[1] Backdoor Attacks on Self-Supervised Learning. In CVPR 2022

W1&Q1. Threat model unclear: dual outsourcing may be unrealistic or costly; performance may drop if models are not independent or only one is used.

A1. We further clarify:

(1) Practicality of the Dual-Model Setting:

• Cost acceptability. Our method requires only two model training sessions (i.e., double the cost) to achieve unified detection without labeled data or clean models. This is more efficient and lightweight compared to existing methods (DECREE, which requires extensive simulated attacks).
• Generality and flexibility. Lie Detector supports multiple architectures and learning paradigms, making it a general detection method. Users can first run Lie Detector on a combination of small models to assess the credibility of service providers at low cost, and then decide whether to use large models, enhancing flexibility.
• Practical feasibility. In security-sensitive industries such as finance, healthcare, and government, cross-model verification and multi-service provider comparisons are already commonplace. Additionally, processes like AutoML and federated learning inherently involve multiple sub-models, which can directly reuse our framework.

A real-world scenario: when a user with a large amount of private data wants to outsource model training, a common practice is to test multiple candidates on a small amount of data (i.e., “competitive bidding”) to evaluate their training capabilities, and then select the optimal one to perform the full-volume training. In this process, the service providers are in competition with each other, and thus the possibility of collusion is low. Our approach can be used for this early evaluation stage, and detect possible backdoor behaviors in the returned models. Once we confirm that a model submitted by a service provider has anomalies, we can blacklist that service provider to avoid subsequent cooperation.

(2) Performance under Model Collusion:

We simulate varying degrees of provider collusion as shown below (“✓”/“✗” means that the two models share/differ that setting):

We report the results (%) below:

Our method remains robust under weak or moderate collusion. Under L1, despite some backdoor data being shared, the models still learn different representations, and CKA can capture these differences. Therefore, Lie Detector remains superior to SOTA under this setting.

As expected, performance degrades under strong collusion (L3), where both models are nearly identical. We have discussed such case in Appendix F. It is worth noting that such strong collusion is extremely rare in practice and difficult to coordinate, especially when the model is trained by independent commercial or decentralized service providers, our method maintains a DSR exceeding 70.0% even in such worst-case scenario.

One potential solution is to randomly distribute the training data to multiple independent training parties, thereby reducing the possibility of collusion from the outset. Ultimately, the models from all parties can be integrated through model fusion (such as ensemble or distillation) to maintain accuracy and achieve robust detection. We plan to make this strategy an important direction for future work and integrated these new settings, results, and analyses into the revised version.

W2. Novelty unclear: combining CKA and fine-tuning sensitivity may appear as a direct application of known techniques.

A2. We clarify that this work is not a simple combination of existing methods, but rather a systematic detection framework for the core challenges of backdoor detection:

• The novelty of the detection framework and the first introduction of CKA. We are the first to apply CKA for backdoor detection. Unlike traditional feature comparisons, CKA better detects training objective shifts. Importantly, we actively optimize triggers to maximize CKA divergence, outperforming other similarity metrics (Appendix D). This mechanism naturally extends to multi-model detection with just two models, offering low cost and easy deployment.
• General detection capability for different learning paradigms. Traditional methods rely on specific architectures, task forms, or loss functions, making them difficult to adapt to different paradigms such as SL/SSL/AL. We combines CKA and OD loss to eliminate such dependencies. We are the first to successfully detect backdoors in multi-modal generative models (LLaVA, MiniGPT-4), achieving a DSR of up to 95.0% (Table 2).
• Fine-tuning sensitivity analysis (FTSA) as a supplementary mechanism. The FTSA is not a simple reuse of fine-tuning algorithms but serves as a behavioral-level supplementary validation mechanism for trigger inversion results. Only when the model indeed embeds backdoor and the trigger is accurately inverted, the $\Delta$ASR will much decrease. If the model is clean or the trigger inversion is ineffective, fine-tuning will have little impact. Therefore, FTSA can enhance detection accuracy while effectively reducing FPR (Table 3).

W3&Q3. Generality concern: different attackers may use different trigger generation methods, which may limit the effectiveness of trigger inversion.

A3. We answer as the following points:

• The goal of the inverse trigger is not to restore the original trigger, but to simulate potential triggerable behavior. This mechanism has been verified in previous work [DECREE][UNICON]. Our method does not rely on restoring the exact trigger, but instead finds a “potential input that can activate abnormal behavior”. This idea has been validated in DECREE, where even if the inversed trigger does not match the real one, the model's abnormal response can still expose the existence of a backdoor.

• The mask + pattern design in equation (4) has high generalization capabilities and can cover multiple types of typical triggers. This design enables flexible representation of different trigger types. This representation has been widely used in various inversion or attack tasks (see DECREE, UNICORN, etc.).To verify the generality of the formula, referring to existing research [2], we mainly divide backdoors into: Static Backdoor Attacks: BadNets, Blended; Content-Aware Backdoor Attacks: LowFrequency, ISSBA.; Model-Adaptive Backdoor Attacks: BadCLIP, TrojanVLM.

• Our experiments have covered various types of triggers, validating the robustness of this method. We evaluate the effectiveness under various attacks in Table 1 and 2 (main paper) and Table 5 (appendix), including (DSR,FPR): BadNets (100.0%, 0.0%), Blended (100.0%, 0.0%), Low-Frequency (100.0%, 0.0%), ISSBA (100.0%, 0.0%), LabelConsistent[1] (95.0%, 0.0%), BadCLIP（95.0%, 5.0%), and TrojanVLM (95.0%,5.0).

[1] Label-Consistent Backdoor Attacks. In arxiv 2019. [2] Revisiting Backdoor Attacks against Large Vision-Language Models from Domain Shift. In CVPR 2025.

W4&Q4. Extra cost concern: if users can't train models, trigger generation may also be too costly; current comparisons may miss this point. Please provide more details about the computation cost.

A4. We will report the average detection time for each model in the paper, and provide comparisons with baselines. TED's runtime is about 6 times longer than ours, particularly on llava, where our runtime is 5.09 minutes, while TED's is 35.48 minutes. For details, please refer to A1 for Reviewer BEna.

Q2. Discuss the effect of using or removing the CKA loss in trigger optimization.

A5. We experiment on removing the CKA loss term Eq. (7).

We see that removing CKA leads to notable drops in DSR and increases in FPR, confirming that CKA loss provides complementary representation-level signals during trigger optimization, especially for complex architectures (CLIP, LLaVA), where output distributions alone may not sufficiently expose backdoor inconsistencies. We will include this in the revised version.

Q5. How to identify that both models are backdoored? There seems lack of such experimental results in the manuscript.

A6. According to Alg. 1 and Appendix A, in the clean-clean case, the CKA value stays relatively high and the trigger has difficulty converging. In contrast, in the backdoor-backdoor and backdoor-clean cases, the CKA value decreases, allowing the reverse-trigger optimization process to converge quickly (there is one trigger per model.) Subsequently, we fine-tuned the model using the reverse triggers and measured the ASR. If the ASR was below a certain threshold, the model was identified as backdoored, otherwise clean. In the experimental setup, the detection results naturally cover all possible cases. On ResNet-18, CIFAR-10, 10 pairs of models were selected for testing under ISSBA attack. The clean-clean, clean-backdoor and backdoor-backdoor detection results are as follows: DSR: 100.0%, 100.0%, 100.0%, FPR: 0.0%, 0.0%, 0.0%.

W1. Hyperparameter sensitivity (e.g., trigger size, canonical strength, fine-tuning steps) is under analyzed. Generalizability to lighter or diverse architectures (e.g., MobileNet, ConvNeXt) also needs further validation.

A1. Thank you for the suggestion. We will explain this in terms of two aspects: hyperparameter sensitivity and lightweight model architecture：

(1) Hyperparameter Sensitivity:

We have conducted a series of ablation studies to analyze the impact of key hyperparameters:

• Trigger mask size: We tested 2×2, 4×4, and 8×8 trigger sizes. Detection results (ResNet-18, CIFAR-10, ISSBA) are summarized below:

 • 2×2 → DSR: 85.0%, FPR: 10.0%

 • 4×4 → DSR: 100.0%, FPR: 0.0%

 • 8×8 → DSR: 100.0%, FPR: 5.0%

• Loss weights (α, β, λ): Moderate changes show limited impact, suggesting robustness of the framework.

• Fine-tuning steps: Performance stabilizes within 10 epochs (DSR: 100.0%, FPR: 0.0%).

• Adaptive thresholding ($\gamma$): We use Gaussian Mixture Modeling (GMM) to fit the distribution of ASR drop values across samples, and automatically select the threshold $\gamma$ that best separates clean and backdoored models to maximize detection performance. For ISSBA on ResNet-18/CIFAR-10, GMM selects $\gamma = 0.22$, yielding 100.0% DSR.

Overall, we can see that our method is not sensitive to those hyperparameters.

(2) Generalizability to Diverse Architectures: We have validated our method across a broad range of models including ResNet-18, VGG16, CLIP, CoCoOp, LLaVA, and MiniGPT-4, covering classification, contrastive learning, and multimodal large models.

To strengthen this further, we include results on lighter models:

• MobileNetV2: DSR: 100.0%, FPR: 0.0%
• ConvNeXt: DSR: 97.5%, FPR: 0.0%

And we can see that oue method can well generalize well to diverse architectures.

W2. Coverage of a typical backdoor attacks (e.g., clean label, multimodal) is insufficient. Current evaluation focuses mainly on trigger-based attacks.

A2. In fact, our paper already includes detection results for multimodal backdoor attacks, specifically:

• TrojanVLM and Shadowcast, two representative attacks on large multimodal models (LLaVA and MiniGPT-4), are evaluated in Table 2.
• Lie Detector achieves the highest detection success rates (e.g., 95.0%, 92.5%, 90.0%) on COCO and Flickr30k, with low FPRs (≤10%), significantly outperforming baselines such as TED, MM-BD, DECREE, and SEER.

Additionally, we have now included results for clean label attacks, specifically Label-Consistent [1]:

We tested the detection performance under clean label attacks on different datasets, and the results further demonstrated the robustness of Lie Detector in complex attack scenarios, especially with a DSR as high as 95.0% on CIFAR10.

[1] Label-Consistent Backdoor Attacks. In arxiv 2019.

W3. Robustness to version differences (e.g., training epochs, parameter initialization) is unclear. Effectiveness under approximate models has not been deeply analyzed.

A3. In real-world deployment, model version differences (e.g., different initialization or training epochs) are common. We have explicitly considered this in both the design and experiments of our method.

(1) Method Design: CKA is Inherently Robust to Version Differences Our method relies on Centered Kernel Alignment (CKA) to measure representation-level similarity, rather than parameter or behavior-level differences. This makes the method robust to variations in initialization and training trajectory. Specifically:

• Orthogonal invariance: CKA remains stable even when different training runs produce activations in different subspaces, as long as the learned semantics are consistent.
• Scale invariance: CKA is unaffected by differences in activation magnitude caused by training length or learning rate.

Thus, models trained with different seeds or epoch counts but similar objectives yield high CKA scores; poisoned models deviate significantly in CKA space due to structural changes in representation.

(2) Experimental Evidence: Stable Detection Under Version Shift
We here include results on model pairs, such as differing in training epochs or parameter initialization.

• Epoch variation (ResNet-18, CIFAR-10, ISSBA):

The 200 vs 100 has the best results, indicating that our detection can improve with sufficient training. Even when one model is partially trained (model1 for 50 epochs), our method remains effective.

• Parameter initialization differences:

We observe minimal performance drop (all above 97.5%) under varying initialization schemes (Kaiming, Xavier, etc.), indicating strong robustness.

In summary, Lie Detector is robust to typical version changes in model deployment, and we will further highlight these analyses in the revision.

W1. It would be better if the paper includes explicit discussion of scalability concerns and computational overhead. Cross-examination involving multiple model trainings and trigger optimization might pose challenges in practice, particularly for large-scale models.

A1. Thank you for pointing out the computational challenges in large-scale settings. Our method achieves consistently strong performance across a range of architectures (from ResNet-18 to MiniGPT-4), and as shown in Table 6, detection accuracy shows no clear negative correlation with GFLOPs.

To further address the concern, we report the average detection time of Lie Detector across different models and datasets ( Followed DECREE and measured on a single NVIDIA A100 80GB GPU.) compared with TED:

Even with large-scale multimodal models, the runtime remains practical (e.g., <4.6 min for MiniGPT-4). We also provide a comparison of detection performance, computational cost, and applicability in Appendix F (Table 6). Lie Detector achieves the highest detection accuracy (99.7%) with substantially lower runtime than TED (92.8% accuracy).

W2. Given the observed sensitivity in experiments (Figure 4), guidance or adaptive methods for parameter selection could strengthen the practical utility.

A2. Thank you for the suggestion. To adaptively determine the detection threshold $\gamma$ in Figure 4, we employ a standard Gaussian Mixture Modeling (GMM) based thresholding strategy. The underlying intuition is that the ASR drop values (i.e., the reduction in Attack Success Rate after applying our defense) follow a bimodal distribution, where clean and backdoored models exhibit distinct statistical behaviors.

Specifically, we fit a two component GMM to the empirical distribution of ASR drop values across all models. Each component is assumed to capture the statistical characteristics of either clean or backdoored models. Once fitted, the GMM provides a probabilistic model of the ASR drop distribution, and we select the threshold $\gamma$ at the intersection point (or the maximum likelihood separation boundary) of the two Gaussian components. This threshold effectively separates clean models (which typically show small ASR drops) from backdoored models (which exhibit larger ASR drops), thereby maximizing the detection performance in a data-driven and adaptive manner.

To more clearly demonstrate the performance changes, we repeated numerous experiments, and this adaptive selection achieved excellent results under different attacks. For example, when using the ResNet-18 model on the CIFAR-10 dataset and subjected to ISSBA attacks, the GMM algorithm identified $\gamma = 0.22$ as the optimal parameter, achieving a detection success rate (DSR) of 100.0% and a false positive rate (FPR) of only 0.0%, which is better than the DSR of 99.61% and the FPR of 0.26% when $\gamma = 0.2$ in Figure 4. Similarly, for low-frequency attacks, GMM selects $\gamma = 0.19$ and achieves a DSR of 99.87% and a FPR of 0.38%, which is better than the DSR of 99.23% and the FPR of 1.03% when $\gamma = 0.2$ in Figure 4. These results demonstrate that GMM-based thresholding generalizes well and often surpasses manually tuned settings.

We will clarify this adaptive procedure and its practical utility in the revised paper.

Q1. What is the runtime of the method on miniGPT-4 (Table 6)?

A3. We will include the runtime for MiniGPT-4 in the revised version. On the Flickr8k dataset, detecting a model pair using Lie Detector takes about 4.53 minutes on a single NVIDIA A100 80GB GPU. In comparison, TED requires 28.19 minutes under the same setting.