JailDAM: Jailbreak Detection with Adaptive Memory for Vision-Language Model

Thank you for acknowledging our contributions. In response to your concerns and potential weaknesses,first, we acknowledge that our model relies on CLIP-based encoders since it doesn’t require task-specific fine-tuning and highly efficient in vision-language tasks. While we use CLIP to encode image-text inputs, our detector can still be applied to both open-weight and black-box (API-only) VLMs, such as GPT-4o-mini (see Figure 4). This design allows our method to generalize across a wide range of VLMs without modification.

On the other hand, our model do have some advantages compared with black-box defense methods such as the Adashield. Adashield-S adopts a single static prompt, which results in relatively lower performance across various models, including Llava-13B, CogVLM-chat-1.1v, and GPT-4o-mini (see Table 5). While Adashield-A employs a dynamic prompting strategy that requires harmful training data for iterative adjustment and incurs significantly higher inference & training time cost. In contrast, our method requires no access to harmful data, achieves stronger or comparable performance across models, and maintains much faster inference & training efficiency.

Overall, our method strikes a balance between generalizability, efficiency, and robustness, making it a practical solution for real-world jailbreak detection.

Thanks for recognizing our contributions in this work. For your concerns and potential weakness, we have the following response:

Methodological Dependence on GPT-4o

• (1) Memory Bank Generation: While we rely on GPT-4o to generate the unsafe memory bank in our implementation, our method is not tied to this specific model. Instead, it provides a generalizable strategy for leveraging LLMs to produce diverse harmful concepts, serving as a scalable alternative to manual curation. To validate the generality of our approach, we additionally generate memory banks using DeepSeek-R1 under the same prompt and do the same experiments. As shown in the performance table below, detectors built on these alternative concept sources yield results that are consistent with our main findings, demonstrating similarly strong performance to the GPT-4o-based setup. This shows that JailDAM is a flexible framework that we can leverage different LLMs.

Table 1: Detection Performance (AUROC / AUPRC) across Different Memory Generators

• (2) Bias of using GPT-4: You raise an important point: relying solely on a single large-scale language model, such as GPT-4o, for harmful memory generation (as in JailDAM) may introduce biases inherent to that model. In our case, we adopted GPT-generated memory primarily due to the lack of a well-defined, standardized harmful content policy. However, this design choice can adapt to real-world settings where different organizations typically enforce their own tailored policies, which helps mitigate the issue of model bias by grounding detection in concrete, context-aware rules rather than relying on a single, potentially biased model’s judgment.

Insufficient Robustness Analysis

Thank you for your valuable feedback. Your suggestion offers a meaningful direction for improving JailDAM. As clarified in the introduction, our work specifically focuses on structure-based jailbreak attacks, which differ from traditional adversarial attacks in both intent and mechanism. While adversarial attacks are indeed an important area of research, they fall outside the current scope of this paper. That said, we agree that exploring both adversarial and structure-based methods could be a promising direction for future work. We will clearly acknowledge this limitation and future direction in the revised paper by adding the following section:

Limitation and Future Work

This study focuses specifically on structure-based jailbreak attacks and does not encompass adversarial attacks such as perturbation-based or gradient-based methods. While this focused approach allows for in-depth analysis of structure-based vulnerabilities, it represents a limitation in the comprehensiveness of our detection framework. Future research will extend this work by developing more generalizable detection mechanisms capable of identifying both structure-based and adversarial attacks within a unified framework. 

We sincerely thank you for the insightful comments and constructive feedback. We agree that the generalization ability of a detector trained solely on safe data and policy constraints can be influenced by the coverage and quality of the conceptual memory bank. To address this concern, we provide a two-part response supported by empirical evidence:

---

1. Memory Comprehensiveness Our first point is that the detector’s ability to generalize across domains benefits significantly from having a comprehensive memory bank. Even when the training and testing domains differ, the model can still capture harmful information because harmful policies are well-covered in the memory bank.

Moreover, even when the memory bank is incomplete, we emphasize that our method is equipped with an Adaptive Memory Mechanism, which mitigates the limitations of incomplete memory coverage. To verify this, we conduct a memory removal experiment on MMSafetyBench dataset. We start with a full memory bank built from the 13 harmful categories defined in MMSafetyBench. We then progressively remove 1, 3, 5, and 7 categories and evaluate the model’s performance. As shown below, the detection accuracy degrades slowly as the memory coverage becomes sparser. This demonstrates that our memory mechanism is robust even when the memory of unsafe policy is not complete.

Table 1: Performance of JailDAM After Removing Memories

2. Out of Domain Generalization to Benign Inputs To directly assess whether the detector generalizes to the out-of-domain test input, we conduct a second experiment using MMMU [1] as a different domain test input. Unlike MM-vet focusing on general VQA, MMMU consists of academic-style content. We compared the difference of 2 datasets as: $\mathbb{E}[\cos(\mathbf{q}_i, \mathbf{q}_j)] \approx 0.11$ where $\mathbf{q}_i, \mathbf{q}_j$ be textual question embeddings. This indicate a substantial semantic shift between MM-vet and MMMU. We then apply the detector (trained solely on MM-vet) to MMMU and evaluate whether it can correctly classify the inputs as safe. The results are summarized below:

Table 2: Out of Domain Generalization of JailDAM

The detector achieves strong performance, demonstrating that our model's robustness and captures generalizable, policy-aligned safety signals.

Together, JailDAM has comprehensive memory bank with adaptive memory, OOD Generalization to Benign Inputs indicating that our detector exhibits strong domain generalization and does not simply memorize training distribution artifacts.

---

[1] Yue, Xiang, et al. "Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.