Backdoor Cleaning without External Guidance in MLLM Fine-tuning

Dear Reviewer jAXd:

Thank you for your careful review and valuable feedback on our work. In the following responses, we address each of your concerns in detail.

W1: Notation Mistake

Thank you for pointing out the inconsistency. In Sec. 3.1, $D_{\text{train}}$ refers to the clean training set for fine-tuning, while in Sec. 4.1, it should be written as $D_{\text{train}} \cup D_{\text{poison}}$, representing the union of clean and backdoored samples. We will revise the notation and clarify the definitions in the updated version.

W2: Discussion on Diverse Trigger

We sincerely thank the reviewer for raising this valuable point. To address this concern, we conduct additional experiments with blended [1] (global, non-patched) backdoor triggers, which better reflect real-world attack diversity. We also include two additional defense baselines, DiffPure [2] and SampDetox [3]. As shown in our new results, BYE still achieves much lower ASR than all baseline methods, confirming that our approach is not limited to simple or visually distinctive triggers, but generalizes well to more adaptive and subtle attack types. This robustness stems from BYE’s focus on behavioral patterns rather than explicit patch localization, allowing it to generalize across trigger types.

In contrast, DiffPure, ZIP and SampDetox reliance on external prior knowledge limits its effectiveness against in-domain or blended triggers, and our results show it performs poorly in these settings.

Table 1: Performance of BYE under the blended trigger attack setting on ScienceQA and IconQA.

[1] Targeted backdoor attacks on deep learning systems using data poisoning. Preprint 2017.

[2] Diffusion models for adversarial purification. ICML 2022.

[3] SampDetox: Black-box Backdoor Defense via Perturbation-based Sample Detoxification. NeurIPS 2024.

W3: Discussion on Computation Cost

Thanks for raising this important concern regarding computational expense! We would like to clarify the actual cost of our method and compare it with alternative backdoor removal approaches.

Computation Cost: Our system BYE consists of three steps: (1) Standard fine-tuning on the (possibly poisoned) dataset (about 30 minutes), (2) One-time inference pass for suspicious sample detection (about 8 minutes), (3) Re-training on the filtered clean set (about 30 minutes). Total: about 72 minutes on 4×4090 GPUs (6k samples). By contrast, diffusion-based methods like ZIP require similar fine-tuning but add a time-consuming purification stage, taking at least 3 hours of GPU time for the same dataset depends on the diffusion model choosed. Total: 162 minutes or more.

Table 2: Comparison of total computational time (in minutes) for BYE and ZIP.

Q1: Discussion on Patch Trigger Visibility

Thank you for these insightful suggestion. We address them as follows:

1. Is the image difference in patch-based attacks typically larger than in adversarial perturbation-based methods?

No. For the model, both patch-based and adversarial triggers function similarly in disrupting internal behavior, regardless of the visual difference. For humans, patch triggers can also appear as real-world objects (e.g., sunglasses), which can be highly inconspicuous.

2. Could simple visual inspection suffice to detect patch-based triggers?

No. While patch triggers are visually perceptible, relying on manual or heuristic inspection is neither reliable nor scalable for real-world, large-scale datasets. Real-world triggers may be any pattern, such as logos or stickers, which are not always obvious to humans. MLLMs do not rely on human-like visual cues; triggers that are obvious to us may not be distinguishable at the model level. Automated, model-based detection methods are thus necessary for practical defense.

Q2: Analysis of Online Filtering

Thank you for this insightful suggestion. The design of BYE as a post-hoc defense is motivated by the need for the model to fully exhibit both clean and backdoored behaviors before effective detection is possible. In our approach, the identification of poisoned samples relies on the model having already learned the backdoor pattern, which allows us to observe clear statistical differences in attention entropy between clean and triggered data. If filtering were performed online during training, BYE would only be able to identify a backdoored sample after the model has already been affected by it. This would lead to a potential dilemma where the model could repeatedly be contaminated, identified, and purified in a cycle, making it difficult to achieve stable and effective defense. By performing detection and purification after fine-tuning, we ensure that the model has fully revealed any backdoor-related behaviors, which can then be accurately distinguished and removed.

Q3: Discussion on Diverse Poison Rate

Thank you for highlighting the concern about GMM clustering on highly imbalanced data. BYE identifies poisoned samples by capturing the behavioral difference between clean and triggered inputs, regardless of the poison rate or class imbalance. As long as the backdoor consistently induces abnormal model responses, the entropy-based feature enables GMM to reliably separate poisoned from clean samples, even when the number of poisoned samples is much smaller.

To verify this, we conducted experiments at 10%, 5%, and 1% poison rates on ScienceQA and IconQA. Results show BYE maintains strong detection and low ASR even with just 1% contaminated data:

Table 3: BYE performance under varying poison rates (10%, 5%, 1%) on ScienceQA and IconQA. In the format CP (↑) / ASR (↓).

Q4: Discussion on Entropy Consistency & Convergence

Thank you for these important questions. We address them as follows:

1. Do entropy patterns from all layers agree?

No, entropy separation is not uniform across all layers. Only a subset of layers show clear discrimination between clean and poisoned samples. Including all layers actually degrades detection. Therefore, we use an adaptive selection strategy, aggregating entropy only from layers with high BSI, as detailed in Appendix A.3. This makes BYE robust to different architectures and tasks.

2. Is BSI separation highly dependent on tuning convergence?

No. Our experiments with checkpoints from different fine-tuning epochs (see table below) show that BYE’s detection effectiveness is stable and not sensitive to the convergence stage. Even with only 1–2 epochs of tuning, the attack success rate remains very low, confirming the robustness of our method throughout the training process.

Table 4: Clean performance and attack success rate (ASR) of BYE at different fine-tuning epochs.

Our adaptive layer selection ensures that BYE focuses on the most informative layers for backdoor detection, avoiding noisy or uninformative ones. Extensive experiments demonstrate that BYE’s effectiveness is robust to the choice of layers and to the degree of model convergence during fine-tuning. These design choices together guarantee that BYE is reliable and practical across different architectures and FTaaS scenarios.

Dear Reviewer ep1A:

We appreciate your engagement with our work and the thoughtful observations you made. We aim to address your concerns in our detailed responses below.

W1&W2&Q1: Comparison with More Baselines

Thank you for raising the concern about the adequacy of our defense baselines. Our comparisons are limited to image purification methods (e.g., ZIP), which do not fully capture the landscape of advanced backdoor defense strategies.

To address this, we add extensive new baselines, including: (1) Neural Attention Distillation (NAD) [1] (attention-based fine-tuning defense). (2) DiffPure [2] and SampDetox [3] (state-of-the-art purification-based methods).

Table 1: Comparison of BYE under the BadNet attack setting on ScienceQA and IconQA.

As shown in Table 1, BYE consistently achieves the best performance across ScienceQA and IconQA, outperforming both attention-based and purification-based defenses. We emphasize that ZIP, DiffPure, and SampDetox are state-of-the-art defenses designed specifically for real-world backdoor threats, and all baselines are evaluated under identical attack types and threat models as BYE.

[1] Neural attention distillation: Erasing backdoor triggers from deep neural networks. ICLR 2021.

[2] Diffusion models for adversarial purification. ICML 2022.

[3] SampDetox: Black-box Backdoor Defense via Perturbation-based Sample Detoxification. NeurIPS 2024.

W3&Q2: Trigger Validity&Diversity Discussion

Thank you for these thoughtful questions regarding our experimental design and the scientific validity of using small black blocks as backdoor triggers.

A1: Using small black squares as visual triggers is a standard and widely adopted practice in the backdoor literature (e.g., BadNets [1], federated learning, and recent CNN-based studies). This setup is favored because it is simple, reproducible, and widely recognized as a baseline, making our results directly comparable to prior work.

A2: To address this, we conduct additional experiments under the more challenging blended trigger attack [2], where BYE still achieves the lowest attack success rate. These results clearly demonstrate the robustness and generalizability of BYE across a range of potential attacks.

Table 2: Performance under blended attack. In the format CP (↑) / ASR (↓).

[1] Badnets: Identifying vulnerabilities in the machine learning model supply chain. Preprint 2017.

[2] Targeted backdoor attacks on deep learning systems using data poisoning. Preprint 2017.

W4: Detailed description in Section 5.5

Thank you for pointing out the need for clarity in Section 5.5. We conduct experiments in Section 5.5 (in page 9, Table 3) specifically to verify the robustness of our method against potential adaptive backdoor attacks. Given that an attacker may attempt to circumvent our detection mechanism by dispersing attention triggers across multiple spatial locations, we design experiments with multi-trigger scenarios, including fixed dual triggers and varied multi-triggers. We aim to demonstrate that our entropy-based detection approach remains effective even when triggers are spatially dispersed or non-localized. To clarify, we implement two adaptive attack scenarios: (1) embedding multiple identical triggers symmetrically placed within a single image (Fixed Dual Trigger); (2) embedding triggers dispersed at regular grid points across images (Varied Multi-Trigger). Results in Table 3 (page 9) confirm our method’s robustness, showing consistent reductions in attack success rates (ASR) and high recall in identifying poisoned samples, thereby validating our defense’s resistance against more sophisticated, dispersed attack patterns. We will expand this discussion in the revised manuscript to explicitly outline the rationale and setup for these experiments.

W5&Q3: Discussion on Computation Cost

Thanks for raising this important concern regarding computational expense! We would like to clarify the actual cost of our method and compare it with alternative backdoor removal approaches.

Computation Cost: Our system BYE consists of three steps: (1) Standard fine-tuning on the (possibly poisoned) dataset (about 30 minutes), (2) One-time inference pass for suspicious sample detection (about 8 minutes), (3) Re-training on the filtered clean set (about 30 minutes). Total: about 72 minutes on 4×4090 GPUs (6k samples). By contrast, diffusion-based methods like ZIP require similar fine-tuning but add a time-consuming purification stage, taking at least 3 hours of GPU time for the same dataset depends on the diffusion model choosed. Total: 162 minutes or more.

Table 3: Comparison of total computational time (in minutes) for BYE and ZIP.

Quantitative comparison with externally supervised methods: In contrast to externally supervised baselines which often require additional annotation, supervision, or computationally heavy denoising modules. BYE’s self-supervised detection incurs only a lightweight, one-time cost and no extra inference latency. The above quantitative comparison demonstrates that BYE is not only more efficient than diffusion-based methods, but also avoids the recurring overhead associated with external supervision.

Q4: Generalization Across Model Architectures

Thank you for this important question about generalizability across model architectures and scales! BYE is designed to be architecture-agnostic: it relies only on extracting cross-modal attention maps, which are present in both decoder-only (e.g., LLaMA, GPT) and encoder-decoder (e.g., T5) models. Its effectiveness does not depend on the number of attention heads, hidden size, or specific attention design. BYE detects distributional shifts in attention caused by backdoor triggers which a phenomenon expected to occur across both types of architectures, since the attack goal is to induce shortcut behaviors in attention, regardless of model structure.

To demonstrate scalability, we conduct additional experiments on LLaVA-1.5-13B (a large decoder-only model) with ScienceQA and IconQA. BYE achieves strong clean performance and robust backdoor mitigation:

Table 4: Clean performance (CP) and attack success rate (ASR) of BYE on LLaVA-1.5-13B, demonstrating robustness and scalability to larger model sizes.

These results show BYE is robust and effective as model scale increases.

Q5: Statistical Significance Analysis

We thank the reviewer for raising the issue of statistical significance. To address this concern, we repeat the main experiments on ScienceQA and IconQA three times with different random seeds and report the mean, standard deviation, and 95% confidence interval (CI) for each key metric in the table below.

Table 5: Statistical analysis of BYE’s main results, demonstrating robustness and stability across multiple runs.

The results show that BYE’s performance is highly stable and robust to randomness, with minimal variance across independent runs. This stability arises because our method is almost fully deterministic: the inference temperature is set to 0, and GMM clustering yields highly consistent results due to the clear separation between clusters.

Dear Reviewer Uw8H:

Thank you for your encouraging remarks and the critical questions you posed. We provide our detailed responses below to address your concern.

W1&Q1: Trigger Diversity Discussion

We thank the reviewer for highlighting this important concern about the attack diversity. Our initial evaluation mainly considered patch-based triggers, we agree it is crucial to test BYE under more distributed and subtle attacks such as blended and reflection triggers [1,2].

BYE remains effective even for global or naturally embedded triggers. No matter if the trigger is a small patch or a blended pattern, the backdoor’s core effect is to teach the model a shortcut so the model shows a consistent, abnormal response pattern. This often leads to a class-consistent, atypical attention distribution, which BYE detects via clustering. While global triggers may not concentrate attention on a single region, they still create distinctive and class-consistent attention shifts compared to clean samples.

To address this, we conducted additional experiments under the blended attack setting and included two state-of-the-art purification-based baselines, DiffPure [3] and SampDetox [4]. The results below show that BYE remains relatively effective and significantly outperforms both baselines under this more challenging scenario:

Table 1: Performance comparison of BYE with DiffPure, ZIP, and SampDetox under the blended trigger attack setting on ScienceQA and IconQA.

[1] Targeted backdoor attacks on deep learning systems using data poisoning. Preprint 2017.

[2] Liu et al. Reflection Backdoor: A Natural Backdoor Attack on Deep Neural Networks. ECCV 2020.

[3] Diffusion models for adversarial purification. ICML 2022.

[4] SampDetox: Black-box Backdoor Defense via Perturbation-based Sample Detoxification. NeurIPS 2024.

Q2: Analysis of Performance Gap on LLaVA&InternVL

Thank you for your insightful question regarding the differing ASR reduction achieved by BYE on LLaVA and InternVL! The main reason for this gap lies in the architectural differences in how these models encode and fuse visual information with language.

LLaVA maintains a relatively high spatial resolution for visual input, representing images with a fixed 24×24 grid (576 tokens) that are linearly projected to match the LLM input dimension (typically 1024). Visual tokens are simply concatenated with text tokens and jointly processed by the LLM, preserving fine-grained spatial cues. This enables BYE to precisely capture attention collapse caused by backdoor triggers and reduce the ASR to near zero.

In contrast, InternVL first encodes images as 1024 visual tokens, but then applies a cross-modal projection to reduce this sequence to just 256 tokens (matching the input dimension of InternLM). Moreover, InternVL employs multiple layers of cross-modal fusion, where visual and language features interact deeply. While this design facilitates stronger multimodal alignment, the lower spatial resolution (256 vs. 576 tokens) means that each visual token encodes information from a larger image region, causing trigger effects to be more diluted or mixed. This leads to a residual ASR around 7. Despite this, BYE still consistently and substantially reduces the ASR for InternVL across all datasets, demonstrating its robustness to architectural differences and lower token resolutions.

Dear Reviewer Bszd:

We are truly grateful for your insightful review and the constructive feedback provided. We respond to each point below with careful consideration.

W1&Q1: Discussion on Computation Cost

Thanks for raising this important concern regarding computational expense! We would like to clarify the actual cost of our method and compare it with alternative backdoor removal approaches.

Computation Cost: Our system BYE consists of three steps: (1) Standard fine-tuning on the (possibly poisoned) dataset (about 30 minutes), (2) One-time inference pass for suspicious sample detection (about 8 minutes), (3) Re-training on the filtered clean set (about 30 minutes). Total: about 72 minutes on 4×4090 GPUs (6k samples). By contrast, diffusion-based methods like ZIP require similar fine-tuning but add a time-consuming purification stage, taking at least 3 hours of GPU time for the same dataset depends on the diffusion model choosed. Total: 162 minutes or more.

Table 1: Comparison of total computational time (in minutes) for BYE and ZIP.

Practical value: Diffusion-based methods introduce significant inference latency and are difficult to deploy in practice, since each test input must be purified at inference time. In contrast, BYE only adds extra time at the provider (server) side during training so users experience no additional delay or difference at inference. The deployed model remains as efficient as a standard model for real-world use.

In summary, BYE’s extra cost is modest, one-time, and entirely server-side. It enables robust backdoor defense without affecting user experience, making the computational expense well justified for practical deployment.

W2&Q2: Trigger Diversity Discussion

Thank you for raising this important concern. We fully agree that classic black square triggers (BadNet), are simple and widely adopted but do not represent the full range of potential backdoor threats.

To address this, we conduct additional experiments using Blended [1] attack, a more global and less visually obvious attack method. As shown in Table 2, BYE still achieves much lower ASR than all baseline methods, confirming that our approach is not limited to simple or visually distinctive triggers, but generalizes well to more adaptive and subtle attack types.

Table 2: Performance under blended attack. In the format CP (↑) / ASR (↓).

To further assess robustness, we also evaluate with diverse triggers. For each poisoned sample, we insert five small black trigger patches at random positions and assign randomly selected target captions from a paraphrase pool. We also compare against two additional defense baselines, DiffPure and SampDetox. As shown in Table 3, BYE effectively reduces ASR to near zero even under these challenging, diverse trigger scenarios, and significantly outperforms all baselines.

Table 3: Performance under diverse trigger and targeted output setting. In the format CP (↑) / ASR (↓).

[1] Targeted backdoor attacks on deep learning systems using data poisoning. Preprint 2017.

W3: Analysis of Innovation Beyond Traditional Defenses

We appreciate the reviewer’s comment. While our method is inspired by traditional backdoor defenses, adapting such strategies to MLLMs presents new challenges:

1. Complex cross-modal attention and modality alignment: MLLMs require handling information across modalities, making direct application of CNN-based methods ineffective.
2. Unsupervised entropy-based detection: Our approach is specifically designed for transformer-based MLLMs, rather than relying on visual or gradient cues.
3. FTaaS compatibility and real-world deployment: The method does not require gradient or data access, making it practical for real FTaaS scenarios.

In summary, our work generalizes and customizes entropy-based backdoor detection for the unique challenges of MLLMs.

Q3: Discussion on Diverse Poison Rate

Thank you for your question regarding the effect of different trigger contamination rates on the effectiveness of BYE.

BYE identifies poisoned samples by capturing the behavioral difference between clean and triggered inputs, regardless of the overall poison rate. As long as the backdoor causes consistent abnormal responses, BYE can reliably detect them.

To verify this, we conducted experiments at 10%, 5%, and 1% poison rates on ScienceQA and IconQA. Results show BYE maintains strong detection and low ASR even with just 1% contaminated data:

Table 4: BYE performance under varying poison rates (10%, 5%, 1%) on ScienceQA and IconQA. In the format CP (↑) / ASR (↓).

These results confirm that BYE remains robust even under extreme conditions such as a 1% poison rate, as long as the backdoor produces a stable effect on the model. The effectiveness of our method is therefore determined by the presence of a detectable behavioral difference, rather than the absolute proportion of poisoned data.

Dear Reviewer wyVd:

We greatly value the time and expertise you invested in reviewing our submission. We address your comments in detail below.

W1: Trigger Diversity Discussion

Thanks for the insightful suggestion regarding the robustness of our entropy-based detection method under more diverse and challenging backdoor attack scenarios.

To address this concern, we conduct additional experiments following the proposed setting: for each poisoned sample, we insert five small black trigger patches at random positions and assign a randomly selected target caption from a paraphrase pool. We also include two additional defense baselines, DiffPure [1] and SampDetox [2].

Table 1: Performance under diverse trigger and targeted output setting. In the format CP (↑) / ASR (↓).

As shown, BYE effectively reduces ASR to near zero on both ScienceQA and IconQA, even under diverse attacks, and significantly outperforms the baselines.

In addition, we further evaluate our method under the blended [3] attack scenario. BYE consistently achieves significantly lower ASR than all baseline methods in this setting as well, further confirming the robustness and generalizability of our approach across various attack types.

Table 2: Performance under blended attack. In the format CP (↑) / ASR (↓).

[1] Diffusion models for adversarial purification. ICML 2022.

[2] SampDetox: Black-box Backdoor Defense via Perturbation-based Sample Detoxification. NeurIPS 2024.

[3] Targeted backdoor attacks on deep learning systems using data poisoning. Preprint 2017.

W2: Upper and Lower Bound Analysis

Thank you for your valuable suggestion! It's important to include results from models fine-tuned on clean samples only, which can serve as an upper/lower bound for CP/ ASR.

In our original Table 1 (page 8), “Vanilla FT” uses mixed clean and poisoned data. Following your advice, we additionally report Clean FT results (no poisoned samples) as shown below:

Table 3: Clean FT vs. Vanilla FT. Clean FT provides the upper bound for CP and lower bound for ASR. In the format CP (↑) / ASR (↓)

As expected, Clean FT consistently yields the highest CP and zero ASR, confirming its role as the upper/lower bound. We will add these results and clarify the different fine-tuning settings in our revised submission.

W3: Entropy Histograms Analysis

Thank you for your valuable suggestion. We perform the experiments and find that the entropy separation between clean and poisoned samples is consistent across Fixed Single, Fixed Dual, and Varied Multi triggers. As we are unable to upload images or external links in the rebuttal, we provide the key sample-level statistics in the table below. The Kolmogorov–Smirnov (KS) distance quantifies the distribution gap for each trigger type.

Table 4: Sample-level attention entropy statistics and KS distances for Fixed Single, Fixed Dual, and Varied Multi triggers.

These results confirm that entropy-based separation is robust for all evaluated trigger types. We will include the empirical histograms in the revised manuscript.