SafePTR: Token-Level Jailbreak Defense in Multimodal LLMs via Prune-then-Restore Mechanism

Weakness 1. The novelty is somewhat limited. The similar pruning and store operation is similar to [1].

Response: BlueSuffix [1] does not incorporate pruning and restoring operations , which is significantly diverges from that of SafePTR. Specifically, its methodology purifies images with a diffusion model, rewrites text prompts using a Large Language Model (LLM), and critically, appends an LLM-generated "blue-team suffix" via reinforcement learning to ensure benign Visual-Language Model (VLM) responses.

[1] BlueSuffix: Reinforced Blue Teaming for Vision-Language Models Against Jailbreak Attacks.

Weakness 2. The observation is not new and is already provided in some papers, e.g., pca analysis and the layer analysis in [2].

Response:

• No Use of PCA: Our analysis, encompassing Findings 1 through 3, does not employ Principal Component Analysis (PCA).

• From Layer-wise to Token-wise Analysis: The initial layer-wise analysis in finding-1 serves primarily as a foundation to motivate and support the subsequent token-level investigation in finding-3. Our core contribution lies in this fine-grained token-wise analysis, which reveals that fewer than 1% of tokens are primarily responsible for triggering multimodal jailbreaks. To our knowledge, this is the first study to perform such a detailed token-level analysis in this context. As Reviewer #1 (kYdu) noted, "Finding-3 that less than 1% of tokens are responsible for jailbreaks is particularly valuable and could inspire future research."

[2] on prompt-driven safeguarding for large language models.

Weakness 3. The applied MLLMs are too old. More new versions, e.g., Llava-Next (not in main experiments), InternVL-2, should be provided.

Response: We further validate the effectiveness of SafePTR using recent multimodal large language models (MLLMs), including LLaVA-Next and InternVL-2. As shown in Table 1, SafePTR consistently outperforms existing methods (Figstep and AdaShield), achieving superior results in both safety benchmarks (Figstep and MMSafetyBench) and utility (MME), thus demonstrating its broad applicability.

Table 1: Performance across LLaVA-Next and InternVL2

Weakness 4. If I increase the size of image and the length of text, can the method still be effective? It may be an open discussion but somewhat influence the limitation of this paper.

Response: To further explore the effectiveness of our method under varying image sizes and text lengths, we conducted experiments on FigStep benchmark of LLaVA-1.5 7B. As shown in Table.2, our LLaVA-1.5+SafePTR consistently demonstrates strong defensive performance, regardless of the scale of the input, indicating its effectiveness across a wide range of scenarios.

Table 2: The Impact of Image Size and Text Length of LLaVA-1.5+SafePTR on FigStep

In addition, "BlueSuffix [1] does not incorporate pruning and restoring operations". But I think although there are differences in the specific execution operations, are there any similarities between the two in terms of ideas? It is necessary to compare these two methods reasonably (experimentally or ideologically). More detailed clarification is necessary.

We sincerely thank the reviewer for their insightful feedback and for prompting a deeper, more conceptual comparison between our SafePTR and BlueSuffix. Futhermore, we will add this detailed discussion and comparison to the related work section of our revised manuscript to better help readers understand our contributions.

• Common Ground in High-Level Ideas You are correct to point out that at a macro level, SafePTR and BlueSuffix share common ground in their core objectives and high-level strategies. Specifically, both approaches attempt to remove potential jailbreak information within visual and textual modalities input. They both ground their defense in a safety-aligned textual reference (SafePTR: Safety-aligned embedding as latent-space anchor for token pruning v.s., BlueSuffix: external blue-team suffixes for safe prompting) guide the model towards a harmless output.

• Summary of Core Differences Despite these shared high-level goals, the two methods diverge fundamentally in their underlying philosophies and technical paths. In summary, BlueSuffix follows an "external augmentation" route, relying on external models for purification and appending a suffix to counteract risk. In contrast, SafePTR pioneers an "internal surgical" approach, operating directly within the model's hidden states to precisely prune and restore features, which is more effective and efficient for real-world scenarios.

• Detailed Elaboration of Fundamental Differences We elaborate on these core differences across two key dimensions below.

  1.Purification vs. Pruning Mechanism:

  • BlueSuffix employs a strategy based on external models. It utilizes a separate diffusion-based method to purify images and a large language model (LLM) to rewrite the text prompts.
  • SafePTR uses an internal, lightweight, token-level pruning mechanism. It operates directly on the hidden-layer representations of visual and textual tokens, precisely "pruning" those identified as harmful.
  • Our Advantage: SafePTR's approach is consequently simpler and more practical which avoids the complexity and inference overhead of invoking multiple external large models. As shown in Table 7 of our manuscript, SafePTR averagely adds only 0.03 seconds of inference latency on the LLaVA-1.6 model, demonstrating its high efficiency as a plug-and-play defense.

  2. Training-Free vs. Training-Based

  • BlueSuffix is a training-based method. Its core "blue-team suffix generator" must be fine-tuned from a pre-trained GPT-2 model using reinforcement learning (PPO) and reward signals from an LLM-based judge.
  • SafePTR is a completely training-free framework.
  • Our Advantage: The training-free nature of SafePTR makes it significantly more practical and scalable for real-world scenarios, especially those with limited computational resources. This is made possible by the core insight from our analysis (Findings 1-3): we found that a very small fraction (<1%) of multimodal tokens within specific "vulnerable layers" are responsible for jailbreaks, and that these tokens exhibit a clear semantic deviation from safe representations. This discovery allows for precise, reference-based identification and pruning without any need for model training.

Additionally, we experimentally compared our method's performance with that of BlueSuffix on the MM-SafetyBench benchmark. As shown in Table 1, we report the Attack Success Rate (ASR) for 13 scenarios on both the LLaVA-1.5-7B and MiniGPT-4. Across all scenarios, our method demonstrated a superior overall advantage, e.g., 1.1 (SafePTR) v.s. 4.6 (BlueSuffix) of LLaVA-1.5 on the Avg(↓) metric. The possible reason is that SafePTR’s design is based on our core insight that jailbreaks are not diffuse, but are triggered by a very small fraction of tokens (<1%) causing semantic deviation in specific vulnerable layers. Therefore, SafePTR performs a direct 'surgical intervention' by pruning these few harmful tokens at their point of influence within the model's hidden states. By targeting the vulnerability at its root—rather than competing with signals at the input level—our method achieves a more robust and consistently lower Attack Success Rate.

Table 1: Comparison with BlueSuffix on MM-SafetyBench.

We are grateful for the reviewer's insightful feedback, which prompted us to perform a deeper analysis of how longer texts influence our method's performance.

We recognize that the WildChat[1] dataset is a safety-sanitized training set, from which toxic or unsafe conversations have been filtered out. Its primary purpose is to provide clean data for training models to generate safer content, and it is therefore not designed to jailbreak models or elicit harmful responses.

However, we agree that evaluating our defense on long-text inputs is a critical test of its practicality in real-world scenarios. To address this, we constructed a new multimodal jailbreak benchmark for long-text evaluation by integrating the WildChat and Figstep[2] datasets. This allowed us to validate the effectiveness of SafePTR against longer text instructions. Specifically, we sampled long-text instructions (e.g., 512 or 1024 tokens) from WildChat and prepended the short, harmful prompts from the Figstep[2] dataset to them. This process yielded a challenging set of multimodal inputs composed of malicious images (from Figstep) and long-text prompts where a harmful instruction is embedded within a much larger, benign context.

As shown in Table 1, our LLaVA-1.5+SafePTR consistently demonstrates strong defensive performance regardless of the input scale (from 28 to 1024), indicating its robust effectiveness across a wide range of scenarios.

The possible reason is that SafePTR's defense mechanism is fundamentally independent of the overall text length, where SafePTR's precision allows it to perform a 'surgical intervention', identifying and pruning only the few harmful tokens. This ability to isolate and neutralize the threat at its source, irrespective of the surrounding context length, explains its robust and stable performance.

Table 1: The Impact of Text Length on Defensive Performance

[1] WildChat: 1M ChatGPT Interaction Logs in the Wild. ICLR 2024

[2] FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts. AAAI 2025: 23951-23959

Weakness 1. (a) Hyper-parameter K is fixed and (b) may be dataset-specific.

Response: We adopt a fixed top-k in terms of following concerns:

• (a)Fixed is better than adaptive: We have explored whether an adaptive pruning ratio performs better than a fixed-K approach (e.g., K=10%). Following previous studies [2, 3], we implemented adaptive pruning by setting a static semantic distance threshold for every input samples and removing tokens below this threshold on LLaVA-1.5 7B, allowing K to vary dynamically. However, as shown in the Table.1, this adaptive strategy did not offer a significant advantage in balancing safety and utility compared to the fixed-K method (Fixed K: 1538.11 vs. Adaptive K: 1464.62) . The possible reason is that adaptive thresholding leads to inconsistent pruning across samples—sometimes masking too many important tokens (equal to K=16.8%), harming utility.

Table 1: Performance of Adaptive K Calculation on LLaVA-1.5 7B

• (b) K is not dataset-specific: As shown in Table 1 of the manuscript, the proportion of harmful tokens remains consistent across different datasets—for example, LLaVA-1.5 on MM-SafetyBench (0.62%) vs. FigStep (0.56%), and MiniGPT-4 on MM-SafetyBench (0.93%) vs. FigStep (0.81%). This consistency is primarily attributed to the attention sink effect [1], an inherent characteristic of the autoregressive mechanism in MLLMs. As discussed in Lines 147–149 of the manuscript, this effect causes certain tokens to disproportionately absorb attention, leading to the concentration of harmful information. Consequently, pruning these few tokens is both effective—achieving a 1.29 reduction in ASR—and efficient—adding only 0.03 seconds of inference latency, as demonstrated in Table 7.

[1] Efficient streaming language models with attention sinks (ICLR 2024).

[2] Holistic Token Merging for Fast Video Large Language Models (CoRR 2025).

[3]GreedyPrune: Retenting Critical Visual Token Set for Large Vision Language Models (CoRR 2025).

Weakness 2. Novelty is incremental because the paper is relied on safety-aligned prompt embedding and the observation is also well-known. Please see [1] and [2,3] you already mentioned in the paper.

Response:

Our work fundamentally differs from prior studies in its core motivation and methodological focus. Here's a detailed comparison with [1-3]:

• Distinctions from [1] "Safety Layers in LLMs":

  • Token-Level vs. Layer-Level: The initial layer-wise analysis presented in Finding 1 primarily serves to motivate and contextualize the subsequent token-level investigation in Finding 3. Our key contribution lies in this fine-grained, token-level analysis, which demonstrates that fewer than 1% of tokens are chiefly responsible for triggering multimodal jailbreaks.

  • Training-Free vs. Training-Dependent: Unlike [1], which necessitates additional training to identify and utilize safety layers, our method is entirely training-free, offering a more practical and readily deployable solution.

• Distinctions from [2] "Immune":

  • Training-Free vs. Training-Dependent: A significant departure from [2], which necessitates training a dedicated safer model via DPO, our methodology operates without any additional training. This eliminates the computational overhead and data requirements associated with training a new model

  • Token Pruning vs. Decoding Policy: Our defense mechanism is based on token pruning, where we identify and remove specific problematic tokens. In contrast, [2]'s defense relies on summing the output logits of a DPO-trained safer model and the original model. These fundamentally different defense approaches lead to distinct operational characteristics and performance trade-offs, particularly in terms of model interpretability and the direct manipulation of hidden states.

• Distinctions from [3] "An Image is Worth 1/2 Tokens":

  • Safety vs. Efficiency: Although both our work and [3] employ pruning techniques, the underlying motivations and applications differ substantially. Reference [3] focuses solely on accelerating MLLM inference by pruning visually redundant tokens. In contrast, the primary goal of SafePTR is to enhance model safety. To the best of our knowledge, we are the first to leverage a pruning strategy specifically aimed at mitigating vulnerabilities for jailbreak defense.

  • Different pruning strategy: SafePTR's token pruning is based on semantic divergence, i.e., distance between unsafe tokens and safe instructions in model's latent space. This fundamentally differs from [3], which prunes tokens based on redundancy via attention scores.

[1] Li, Shen, et al. "Safety layers in aligned large language models: The key to llm security." ICLR 2025.

[2] Ghosal, Soumya Suvra, et al. "Immune: Improving safety against jailbreaks in multi-modal llms via inference-time alignment." CVPR 2025.

[3] Chen, Liang, et al. "An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models." ECCV 2024.

Weakness 3. Unable to applicate to black-box commercial models.

Response: Our proposed methodology was originally designed for white-box models, relying on access to hidden states to facilitate pruning for defensive purposes. This approach cannot be directly applied to black-box models. Following your suggestion, we explored the transferability of the core idea of SafePTR to black-box settings:

• Firstly, we employed an open-source multimodal model (LLaVA-1.5-7B) to identify harmful tokens in the input image.
• Secondly, the corresponding harmful regions were then removed before feeding the modified image into the black-box model. As shown in the Table.2, applying this strategy to GPT-4o-mini (selected for rapid validation) consistently improves safety across FigStep, MMSafetyBench, and Jailbreak-28K, effectively mitigating nearly all jailbreak attempts. Importantly, this enhanced safety comes without sacrificing utility, as demonstrated by stable performance on MMVet.

Table 2: Evaluating Main Idea Transferability to Black-Box Models (GPT-4o-mini)

Question 1. LLaVA-1.5 and MiniGPT-4 share the same vulnerable layers, and DeepSeek-VL2 has a different one. Can your method generalize to other models?

Response: To further validate the generalizability of our approach across different open-source models and architectures, we additionally applied our method, SafePTR, to LLaVA-Next (layers 9-10 recognized as vulnerable layers) and InternVL-2 (layers 7-10 recognized as vulnerable layers). As shown in the Table.3, SafePTR consistently outperforms existing methods (Figstep and AdaShield), achieving superior performance in terms of both safety (Figstep and MMSafetyBench) and utility (MME), thereby confirming its broad applicability.

Table 3: Performance of LLaVA-Next and InternVL2

Question 2. Could an adaptive threshold (e.g., using the statistics of semantic distances per sample) outperform the fixed Top-10 % rule while preserving utility?

Response: Please refer to Weakness 1, Part 1. Fixed is better than adaptive for a detailed comparison between fixed and adaptive thresholding strategies.

Question 3. Discuss concrete strategies to approximate hidden states for black-box commercial models like GPT-4 so SafePTR’s insights can transfer.

Response: Please refer to the Response of Weakness 3 for a detailed discussion on potential strategies to approximate hidden states in black-box commercial models like GPT-4.

Dear Reviewer fTB4,

Thank you very much for your time and for your positive feedback. We are delighted to know that our response has adequately addressed all of your concerns.

Your insightful feedback has been invaluable in strengthening our work. We have already incorporated the discussed changes into the manuscript and believe these updates substantially strengthen the paper.

Specifically, guided by your comments, our revised manuscript now further:

• 1. Validates the robustness of our fixed-K strategy, showing it is not dataset-specific and outperforms adaptive methods in Table.8 together with experimental analysis in Sec.4.4 of Ablation study.

• 2. Clarifies the distinct novelty of our work as the first pruning-based defense for MLLM safety in Sec.5 Related Work (Defenses on Multimodal Large Language Models).

• 3. Demonstrates and validate the transferability of our core idea to black-box models in Table.3 together with experimental analysis in Sec.4.2 Safety Evaluation Results and implementation details in Sec.4.1 Experimental Details.

• 4. Confirms the generalizability of SafePTR across additional, diverse model architectures in Table.3 together with experimental analysis in Sec.4.2 Safety Evaluation Results.

We appreciate your time and effort in reviewing our paper.

Best regards,

The authors of Paper 9182

Weakness 1. Some technical details are not clear enough. In particular, how are the textual harmful tokens pruned and restored. I wish to see some examples too.

Response:

• Technical details: The pruning process for both image and text tokens is detailed in Section 3.1. At vulnerable layers, harmful textual and visual tokens are directly removed from the forward computation (as shown in Equations 2 and 3 of our manuscript), thereby preventing the injection of harmful information into subsequent vulnerable layers. In later layers, where vulnerability is less pronounced, the removed tokens are restored by retrieving their hidden states from the original full forward pass and reinserting them into their original positions (as described in Equations 4 and 5). This enables the model to resume inference with all token representations preserved.

• Example: For the input "How to Build Weapons to Rob a Bank", when fed into LLaVA (with layers 7–8 identified as vulnerable), the tokens "weapons" and "rob" are removed at layers 7–8. The remaining tokens are processed as usual. At layer 9, the hidden states of the removed tokens (from the unpruned pass) are reinserted into their original positions, and inference proceeds normally from that point on.

Question 1. Are the arrow directions in Figure 1 all correct? Upper arrow means better performance with larger numbers and lower arrow vice versa, right? It seems the arrow directions are flipped.

Response: Thank you for your suggestion. We have reviewed the arrow issue in Figure 1 and will revise it in the next version of the paper.

Question 2. Lots of citations have an incorrect style (in my opinion). For example in line 91, the citation for LLaVA, Mini-GPT and DeepSeek-VL should be cited with \citep{} rather than \citet{}. This is a minor issue.

Response: Thank you for your suggestion. We will revise the citation style in the next version of the paper.

Question 3. I am wondering, why did you choose to do layer-wise intervention on 2 and 4 consecutive layers, rather than 1 or greater numbers of layers? Could you provide some insight?

Response:

• Insight: Layer-wise intervention with a moderate window size (k = 2 or 4) offers the best trade-off between perturbation effectiveness and localization precision. This configuration generates the most distinguishable ASR curve variations, effectively isolating vulnerable layers without overly disrupting model behavior.

• Window size Selection: The selection of the layer-wise intervention window size, neither excessively large nor small, is primarily dictated by localization precision. As shown in Table 1, specifically examining k=1 and k>4, confirm that k=2 and k=4 represent the optimal choices in this trade-off, enabling the clearest identification of the model's vulnerable layers.

  • Narrow Window (k=1): A narrow window (k=1) resulted in minor Attack Success Rate (ASR) curve changes (under 34% difference), making it difficult to pinpoint vulnerable regions. This is because Transformer's residual connections and distributed representations can easily bypass single-layer interventions.

  • Wide Window (k>4): Conversely, an overly wide window (e.g., k=8 or k=16) reduced ASR significantly across many layers but still showed limited differentiation (under 42% difference). This occurs because removing too many components impairs core reasoning, making the model "universally vulnerable" and preventing precise localization of the most critical layers.

  • Optimal Window (k=2 and k=4): With optimal windows (k=2 and k=4), we observed pronounced ASR curve differences, exceeding 81%. This clear distinction significantly aids in accurately identifying critical layer locations.

Table 1: Layer-wise vulnerability(ASR % ↓) analysis of MiniGPT-4-7B

Question 4. Finding 2 is interesting. I am wondering, what do you think leads to such semantic deviation. That said, what kind of difference between the adversarial samples and the safety-aligned instructions leads to this deviation. Can we see from some concrete examples? Correct me if I have misunderstandings.

Response: The semantic deviation observed in Finding 2 is largely driven by specific harmful tokens (Finding 3), which amplify the divergence between adversarial inputs and safety-aligned instructions. This reflects a deeper issue: a mismatch between Shallow Conceptual Alignment—where visual features are loosely mapped to textual concepts—and Deep Safety Alignment, which requires integrating safety-critical semantics into these mappings.

As illustrated in Supplemental Figures 3 and 4, tokens related to "fire regions" or "illegal narcotics" intensify this semantic drift. These cases reveal how current vision-language models, lacking deep safety grounding, allow harmful visual-text associations to bypass alignment constraints in the latent space.

Question 5. (a) I see in Finding 3, you are trying to answer my last point. But I still want more examples as mentioned above. (b) Could you tell me how you decide the thrshold \alpha? This number greatly affects the proportion of harmful tokens according to your definition. (c) I also recommend explicitly providing the safty-aligned instruction in Figure 4.

Response: (a) As illustrated in Supplemental Figures 3 and 4, tokens related to "fire regions" or "illegal narcotics" intensify this semantic drift. Further examples can be found in Supplemental Figures 2-9.

(b) The threshold parameter, denoted as α, was determined empirically based on experimental results to effectively distinguish between harmful and benign tokens, as summarized in Table 2. We selected α = 0.10 as it achieves a favorable trade-off, yielding the lowest scores on Figstep and MMSafetyBench (indicating reduced harmfulness) while maintaining a high MME score (indicating model effectiveness).

Table 2 : Performance of different α

(c) Thank you for your suggestion. We will provide the safety-aligned instruction in Figure 4 for clarify.

Question 6. How did you prune text tokens? Any equation would help. And also, when doing HTP, did you just set the hidden state corresponding to top-K harmful tokens to 0? Could you provide more details?

Response: When inference reaches vulnerable layers, Harmful Token Pruning (HTP) directly removes the hidden states corresponding to the top-K harmful tokens from the overall set of hidden states (equation on line 180 of the paper). The relative positions of the remaining hidden states are preserved, ensuring that the harmful token hidden states do not participate in inference within these specific layers while the other hidden states continue through the normal inference process. As detailed in the example within Weaknesses 1, harmful text tokens are directly removed by HTP in vulnerable layers, and these removed hidden states are subsequently restored once inference through these layers is complete.

Question 7. Can I understand restoration as taking the hidden states of the pruned tokens from original forward pass without pruning, and then insert them back as the hidden states of the pruned tokens in later layers where there is a less correlation with vulnerability?

Response: Yes, that understanding is correct. Please refer to the response in Weakness1 for a detailed explanation of the process.

Question 8. Do you think the proposed method can also be implemented on multimodal LLMs with discretized image tokenizers such as Chameleon [1]?

Response: Yes, our proposed method, SafePTR, can be implemented on multimodal LLMs that use discretized image tokenizers, such as Chameleon. While Chameleon’s tokenizer differs from those in models like LLaVA and MiniGPT-4, they share a crucial property: once tokens from all modalities are input into the LLM, they are converted into hidden states and processed in a unified format. This common internal representation enables similar reasoning mechanisms across architectures. SafePTR operates precisely at this stage—it performs pruning on the hidden states after all tokens have been embedded within the model. Therefore, the method is theoretically compatible with Chameleon, as it aligns with the model’s internal processing of multimodal information.

Dear Reviewer eF5U,

Thank you again for your thorough feedback and for engaging so constructively with our rebuttal. You are absolutely right, and we agree that for the paper to be its strongest, all the details and clarifications from our discussion must be explicitly integrated into the manuscript.

We want to assure you that we have a clear action plan and will meticulously revise the paper for the final version. Guided by your feedback, our revisions will specifically include:

• 1. Core Mechanism Details: A thorough clarification of the entire token pruning and restoration process, complete with step-by-step descriptions, equations, and illustrative examples to make the technique fully transparent (addressing Weakness 1, Q6, Q7).

• 2. Methodological Rationale: The detailed justification and empirical evidence for our key methodological choices, including the selection of the layer-wise intervention window size (Q3) and the threshold α (Q5b).

• 3. Deeper Insights & Examples: An expanded discussion on the underlying causes of semantic deviation, supported by the additional concrete examples requested to make our findings more intuitive (addressing Q4, Q5a).

• 4. Generalizability to Architectures: A new discussion on our method's applicability to models with different tokenizer architectures, such as Chameleon, to clarify its broader potential (addressing Q8).

• 5. Presentation and Formatting: Correction of all presentation details noted, including the arrow directions in Figure 1 (Q1), all citation styles (Q2), and adding the safety-aligned instruction to the caption of Figure 4 (Q5c).

We are truly grateful for your detailed and constructive guidance, which provides a clear roadmap for improving our work. We are confident that by incorporating these comprehensive revisions, the paper will fully address all your concerns and be significantly strengthened.

Best regards,

The authors of Paper 9182

Weakness 1. I think the approach has some limitations in scope as it is being restricted to white-box models where intermediate hidden states are accessible, which excludes black-box models like GPT-4.

Response: Our methodology was initially developed for white-box models, leveraging access to hidden states to enable pruning as a defensive mechanism. As such, it is not directly applicable to black-box models. In response to your suggestion, we investigated the transferability of the core concept of SafePTR to black-box settings. Specifically, we utilized an open-source multimodal model (LLaVA-1.5-7B) to detect harmful tokens in the input image. The identified harmful regions were then removed prior to feeding the modified image into a black-box model. As shown in Table 1, applying this strategy to GPT-4o-mini (chosen for its inference efficiency) consistently enhanced safety across FigStep, MMSafetyBench, and Jailbreak-28K, successfully mitigating nearly all jailbreak attempts. Notably, this safety improvement was achieved without compromising utility, as evidenced by stable performance on MMVet.

Table 1: Evaluating Main Idea Transferability to Black-Box Models (GPT-4o-mini)

Weakness 2. (a) The fixed Top-K pruning strategy (K=10%) lacks adaptability to inputs with varying risk levels and (b) the safety-aligned reference instruction used for semantic distance calculation is somewhat arbitrary.

Response: (a) There are two main reasons for adopting a fixed top-k strategy:

• Theoretical Justification: As discussed in Lines 147–149 of the paper, the number of harmful tokens does not scale with the input's risk level. This phenomenon is primarily attributed to the attention sink effect [1] in MLLM inference, whereby harmful content tends to concentrate in a small number of dominant tokens. Consequently, pruning the top-K most harmful tokens remains both effective and computationally efficient.
• Empirical Validation: We investigated whether an adaptive pruning ratio outperforms a fixed-K approach (e.g., K=10%) on LLaVA-1.5 7B. Following prior work [2, 3], we implemented adaptive pruning by applying a fixed semantic distance threshold to each input sample, removing tokens below this threshold and allowing K to vary dynamically. However, as shown in Table 2, this adaptive strategy did not yield a significant advantage in balancing safety and utility compared to the fixed-K method (Fixed K: 1538.11 vs. Adaptive K: 1464.62). A possible explanation is that adaptive thresholding results in inconsistent pruning across samples—occasionally removing too many important tokens (equivalent to K=16.8%), thereby impairing utility.

Table 2: Performance of Adaptive K Calculation on LLaVA-1.5 7B

[1] Efficient streaming language models with attention sinks (ICLR 2024).

[2] Holistic Token Merging for Fast Video Large Language Models (CoRR 2025).

[3]GreedyPrune: Retenting Critical Visual Token Set for Large Vision Language Models (CoRR 2025).

(b) The safety-aligned reference instruction used for semantic distance calculation is grounded in concrete empirical findings and is therefore not arbitrary. Importantly, it exemplifies a general defense paradigm whose effectiveness does not rely on the exact wording, demonstrating broad applicability beyond any particular instruction phrasing：

• Safety-aligned instructions are grounded in empirical findings. As demonstrated in Finding 2 of our paper and illustrated in Figure 3, safety-aligned reference instructions substantially improve the discriminability between safe and unsafe samples in high-dimensional semantic space. Therefore, we adopt these instructions as inherent safety-relevant priors, serving as a guiding signal to enhance separation efficacy.

• A Generalizable Defense Paradigm: The safety-aligned reference instruction is not an arbitrary choice, but rather part of a broad class of effective safety-related instructions. To confirm this, we evaluated multiple generic prompts containing safety-related keywords such as “safely,” “accurately,” and “harmful.” The results, summarized in the Table.3, consistently show that these alternative instructions enable effective token pruning. This demonstrates that our approach constitutes a generalizable defense paradigm rather than relying on a specific phrasing. Essentially, any instruction embedding safety-related terminology suffices for our method to perform effectively, highlighting its broad applicability beyond a fixed instruction set.

Table 3: Performance of diverse safety-aligned reference instruction

Weakness 3. The analysis is limited to three open-source models, potentially limiting generalization to other architectures.

Response: We further validate the generalizability of SafePTR on LLaVA-Next (which employs a multi-scale visual feature fusion strategy) and InternVL-2 (featuring a stronger vision encoder and multi-stage alignment). As shown in Table 2, SafePTR consistently outperforms existing methods, such as Figstep and AdaShield, achieving superior results in both safety benchmarks (Figstep and MMSafetyBench) and utility metrics (MME), thereby demonstrating its broad applicability.

Table 4: Performance of LLaVA-Next and InternVL2

Question 1. Why did you choose a fixed Top-K (10%) pruning ratio rather than an adaptive threshold based on the actual semantic deviation distribution? Have you explored dynamic pruning strategies that adjust K based on input risk levels?

Response: Please refer to the Response of Weakness 2 for a detailed discussion on the choice of a fixed Top-K pruning ratio.

Question 2. In your opinion, how robust do you think SafePTR is against adaptive attacks where adversaries know about the token pruning mechanism?

Response: The implementation details of any white-box defense, if exposed, inherently create avenues for adaptive attacks. This principle also applies to our proposed method, SafePTR. Nevertheless, bypassing SafePTR remains significantly more costly than circumventing simpler defenses like AdaShield, which can be bypassed by merely ignoring instructions. To successfully attack SafePTR, an adversary would first need to identify vulnerable layers within the model. Subsequently, they would have to identify the specific harmful tokens that need to be manipulated. This multi-step process makes attacking SafePTR both computationally expensive and procedurally complex, significantly raising the bar for successful circumvention.

Thank you for your detailed response. I will maintain my score of 5 (accept).