VLMGuard: Defending VLMs against Malicious Prompts via Unlabeled Data

We thank you for recognizing our work as highly intuitive and for studying an important challenge. We appreciate the reviewer's comments and suggestions, which we address below:

A1. Emphasis on unique insights

Great point raised! While it is true that SVD has been used in anomaly detection tasks, VLMGuard’s application and enhancements in the context of VLMs and malicious prompt detection introduce several novel insights:

• Multimodal Context: Unlike prior works that apply SVD to unimodal datasets (e.g., tabular or image data), VLMGuard is the first work to specifically targets the multimodal embeddings of vision-language models for malicious prompt detection. These embeddings encode complex interactions between vision and text inputs, requiring tailored approaches to effectively separate benign and malicious prompts. Malicious prompts in VLMs are inherently different from conventional anomalies. They exploit vulnerabilities in both textual and visual modalities, which makes their detection more challenging than uni-modal detection (see parapgraph 1 in introduction). VLMGuard’s SVD-based analysis identifies a subspace capturing these multimodal adversarial patterns, which is a novel application of SVD.
• Unsupervised Framework: Unlike traditional anomaly detection tasks that often rely on labeled anomalies for validation, the core contribution of VLMGuard is that it provides an innovative approach that operates entirely on unlabeled data without costly human annotation on the maliciousness of the prompts. This necessitated the development of a novel approach to use SVD effectively in this unsupervised setting, including careful handling of mixed distributions of benign and malicious prompts.
• Detailed Method Design: In addition to SVD, our learning framework includes the calculation of the maliciousness score, derived from projections onto the SVD-identified subspace for maliciousness estimation. Finally, the scores are used to train a binary safeguarding prompt classifier. These novel technical contributions along with the comprehensive experimental evaluations on flagship VLMs and representative multimodal threat models offer valuable insights to the LM safety research community.

A2. Clarification on the binary classifier

Another great point! We are happy to clarify.

• Firstly, we would like to point out that our results in Figure 4 have confirmed that training an additional prompt classifier can indeed benefit the detection performance compared to directly thresholding using the maliciousness scores calculated on the unlabeled data. This provides practical justification.
• Secondly, we would like to highlight the technical benefits of the classifier training. Intuitively, the maliciousness score, defined as the norm of projections onto the principal subspace (Eq. 7), effectively captures the degree of alignment with the subspace identified by SVD. However, this method assumes a simple threshold-based separation between benign and malicious samples. When the distributions of benign and malicious samples overlap (as shown in Appendix B, Figure 6), a linear threshold cannot optimally separate the two classes. In contrast, the binary classifier starts with the noisy labels derived from the maliciousness score but has the flexibility to learn a non-linear decision boundary that accounts for additional separability cues present in the full representation space. For instance:
  • It can exploit higher-order relationships or interactions between dimensions in the activation space that the score-based projection cannot capture.
  • It can adjust for systematic noise or bias in the score-based annotations by finding a boundary that better fits the true underlying structure.
• From a theoretical perspective, noisy labels can still lead to optimal classifiers under certain conditions [1], such as when the noise is unbiased or the classifier has sufficient capacity to model the true decision boundary. While the maliciousness score measures alignment with the top singular vectors, it inherently ignores information outside the identified subspace. The classifier, trained on the full representation, has access to the remaining latent dimensions, which may contain complementary information useful for distinguishing malicious prompts. Moreover, the training process of the binary classifier can be viewed as a denoising operation, where the model learns to generalize from the noisy score-based labels to better approximate the true underlying data distribution. This enables it to achieve performance beyond the initial noisy annotations.

We believe further rigorous analysis by exploring noisy label learning literature can be a promising future direction! Thank you pointing this out!

The author's response is consistent with my understanding. So, I have a significant doubt: First, the "maliciousness estimation score" is estimated, and then different positive or negative labels are assigned to the samples based on a threshold. A binary classifier is trained using these labeled data, and this classifier is then used to label unseen data. Since this is the case, why not directly label the unseen data based on the maliciousness estimation score? In other words, the maliciousness estimation score already serves as the ground-truth label for the binary classifier.

We are deeply encouraged that you recognize our method to be simple and effective and give a good reference to the research on this aspect. We appreciate the reviewer's comments and suggestions, which we address below:

A1. Clarification on performance of the binary classifier

You raise a great point! The discrepancy between the binary classifier’s performance and the direct use of the maliciousness score stems from the classifier’s ability to leverage additional representation information and refine decision boundaries beyond the initial score thresholding. Based on this, we provide our detailed reasonings below:

• Intuitively, the maliciousness score, defined as the norm of projections onto the principal subspace (Eq. 7), effectively captures the degree of alignment with the subspace identified by SVD. However, this method assumes a simple threshold-based separation between benign and malicious samples. When the distributions of benign and malicious samples overlap (as shown in Appendix B, Figure 6), a linear threshold cannot optimally separate the two classes. In contrast, the binary classifier starts with the noisy labels derived from the maliciousness score but has the flexibility to learn a non-linear decision boundary that accounts for additional separability cues present in the full representation space. For instance:
  • It can exploit higher-order relationships or interactions between dimensions in the activation space that the score-based projection cannot capture.
  • It can adjust for systematic noise or bias in the score-based annotations by finding a boundary that better fits the true underlying structure.
• From a theoretical perspective, noisy labels can still lead to optimal classifiers under certain conditions [1], such as when the noise is unbiased or the classifier has sufficient capacity to model the true decision boundary. While the maliciousness score measures alignment with the top singular vectors, it inherently ignores information outside the identified subspace. The classifier, trained on the full representation, has access to the remaining latent dimensions, which may contain complementary information useful for distinguishing malicious prompts. Moreover, the training process of the binary classifier can be viewed as a denoising operation, where the model learns to generalize from the noisy score-based labels to better approximate the true underlying data distribution. This enables it to achieve performance beyond the initial noisy annotations.

Based on these reasonings, we believe that the upper bound of the binary classifier's performance is not necessarily limited by the accuracy of the maliciousness score detection. However, we agree with the reviewer that they should be related under certain conditions. Further rigorous analysis by exploring noisy label learning literature can be a promising future direction!

A2. Experimenting with higher maliciousness ratio

Absolutely! We intend to choose a small maliciousness ratio in order to simulate the real-world settings where the majority of input prompts are benign while only a small fraction of them are malicious. As suggested, we follow the same experimental setting as in Figure 3 (b), provide the experimental results when $\pi=0.3, 0.5, 0.7, 0.9$, and report the detection performance as follows. The results achieve near optimal performance and plateau when $\pi$ becomes larger. This precisely highlights that our approach only requires a small number of malicious samples for training a perfect binary malicious prompt classifier, which aligns with the practical application scenarios.

[1] Natarajan et al., Learning with Noisy Labels, NIPS 2013.

We are glad to see that the reviewer recognized our work solve an increasingly important safety issue and the approach to be effective and interesting. We thank the reviewer for the thorough comments and suggestions. We are happy to clarify as follows:

A1. Clarification on the motivation

Thank you for pointing this out! We are happy to clarify.

Motivation. Firstly, as illustrated in the first two paragraphs of introduction of our paper, the motivation for VLMGuard arises from the unique vulnerabilities of VLMs, which operate on multimodal inputs (text and vision) and integrate cross-modal information. Compared to unimodal models, VLMs face greater challenges in detecting malicious prompts because such prompts can exploit vulnerabilities in either modality or their interactions. Specifically, malicious prompts in VLMs often manifest as subtle adversarial perturbations to visual components or as carefully crafted textual instructions that disrupt the model's alignment. VLMGuard is designed to tackle these challenges by leveraging the distinctive properties of VLM embeddings, which encode multimodal contextual information to detect such malicious inputs effectively.

VLM concepts in VLMGuard. The key component of VLMGuard is its ability to analyze the latent subspace in the VLM's embedding space. The embeddings reflect multimodal alignment, which can capture the nuanced interactions between vision and text inputs. Unlike generic deep neural networks, VLMs integrate information across visual and textual modalities. As such, applying VLMGuard to a non-VLM model would require a fundamentally different interpretation of the latent subspace. VLMGuard operates directly on these cross-modal representations, and utilize their specific structure to identify anomalies indicative of malicious intent. During rebuttal, we empirically verify that using the multimodal embeddings shows better malicious prompt detection performance compared to uni-modal representations within the same model (comparison in the below table on detecting adversarial meta-instruction for LLaVA).

Finally, our maliciousness estimation score leverages the multimodal alignment property to identify patterns specific to multimodal malicious prompts, such as adversarial meta-instructions and jailbreak attacks. In fact, our paper is the first work to study detecting different kinds of malicious prompts designed for the multimodal domain on prominent VLMs (e.g., LLaVA and Phi-3). These experiments demonstrate VLMGuard’s effectiveness in addressing VLM-specific threats, which we believe can reinforce its design focus and make our method a significant contribution to the research community.

VLMGuard is not just a general binary classifier. While VLMGuard includes a binary classification step, its core contribution lies in leveraging the unique properties of the unlabeled data and their VLM embeddings to estimate maliciousness. The subspace decomposition step is tailored to capture the high-dimensional relationships and multimodal alignment inherent to VLMs, which makes it less directly applicable to generic neural networks. In contrast, general deep neural networks typically lack the structured multimodal embeddings present in VLMs. Therefore, applying VLMGuard to a non-VLM model would require a fundamentally different interpretation of the latent subspace.

Thank you again for bringing this up!

A2. Presentation issues

Thank you for the question! We would like to refer the reviewer to paragraphs 3 and 4 in the introduction for the description of our proposed approach. In paragraph 3, we introduce the unlabeled modeling framework as an important problem setup. Subseqently, we focus on explaining the key components of the proposed learning method about how to utilize the unlabeled data for malicious prompt detection. We believe that is helpful for readers to understand our method.

For Figure 1, we have included the annotations of the key concepts in our learning framework in the picture and then described the algorithm details in the caption.

If you have any specific details that you want us to clarify and revise for the figure and introduction, please do not hesitate to leave your comment!

A3. Discussions on the geometric intuition

You raise a great point! To the best of our knowledge, the rigorous analysis of our geometric intuition, i.e., how does the LLM representations encode knowledge is still an open-ended research question in LLM Interpretability research domain, and leads to multiple recent papers working on this [5-7]. That being said, we are more than happy to provide our own understandings on your questions!

Firstly, we hypothesize the intuition that "malicious samples may occupy a small subspace" is based on the assumption that malicious prompts represent rare and distinct patterns compared to benign prompts in the training data. This might be theoretically grounded under the condition of the low mixing ratio ($\pi$). In practice, the language model (LM) is trained with much more benign texts over malicious texts to perform general NLP tasks. Therefore, in addition to encoding the discriminative information for benign/malicious separation in its first several principal component of the representation space, LM representation subsumes more information in order to perform the tasks like text continuation, instruction following and reasoning. Such information can include the sentiment, grammar, lexical semantics, etc.

Relation to the VLM's training objectives. Since VLMs are trained to align representations of text and image inputs, often through contrastive objectives or cross-modal attention. These objectives encourage embeddings of benign prompts to occupy a smooth and coherent manifold in the representation space. Malicious prompts, designed to exploit weaknesses in these alignments, introduce outlier patterns that deviate from this manifold, naturally forming a separate subspace.

During training, VLMs are predominantly exposed to benign samples. This leads to a strong alignment of benign prompts with the main directions of variance in the representation space, while malicious prompts, being unseen or adversarially crafted, disrupt this alignment and are projected into distinct regions.

Moreover, malicious prompts often exploit gradient-sensitive directions in the model’s loss landscape (e.g., through adversarial perturbations). These sensitive directions tend to correspond to sparse, high-variance subspaces in the activation space, which our SVD-based method effectively identifies.

[1] https://en.wikipedia.org/wiki/Principal_component_analysis

[2] Einsiedler et al., Functional Analysis, Spectral Theory, and Applications

[3] Li et al., Inference-Time Intervention: Eliciting Truthful Answers from a Language Model, NeurIPS 2023

[4] Zou et al., Representation Engineering: A Top-Down Approach to AI Transparency, 2023

[5] Zheng et al., On Prompt-Driven Safeguarding for Large Language Models, ICML 2024

[6] Friedman et al., Interpretability Illusions in the Generalization of Simplified Models, ICML 2024

[7] Marks et al., The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets, COLM 2024

We thank the reviewer for the comments and suggestions. We are encouraged that you recognize our problem definition to be new and practical and with an efficient and robust technical approach. We address your questions below:

A1. Clarification on the theoretical foundation

Thank you for the suggestion! While the focus of the submission is not on theory, we do think it is helpful to derive a rigorous understanding of our approach. We provide our reasonings as follows:

• Firstly, we believe that the theoretical basis for using the SVD subspace analysis stems from foundational properties of singular value decomposition (SVD) as a dimensionality reduction method. Specifically, SVD identifies dominant modes of variation in data [1], which can encapsulate structural features of embeddings. In our case, we hypothesize and empirically demonstrate that malicious prompts often induce unique patterns in the representation space, which are effectively captured as primary modes of variation by SVD. This separability is supported by the distribution analysis in Figure 6 (Appendix B), where malicious prompts exhibit distinct projection magnitudes.

  To strengthen the theoretical grounding, we believe the exploration on spectral analysis [2] of the covariance matrix on the unlabeled data can be a promising next step. Specifically, malicious samples can be modeled as perturbations of the benign samples. Therefore, we could leverage the perturbation analysis theory to quantify the difference in eigenvalues between the top-$k$ components and the rest to assess the contribution of malicious prompts. A formal treatment of this hypothesis is a natural extension of our work.

• In terms of the generalization across different attacks, spectral analysis could be a useful theoretical analysis tool as well. For example, a simple starting point for analysis is to model the different attacks as a distribution drawn from a proxy distribution, such as Gaussians. Based on this proxy modeling, we could subsequently analyze the spectral gap between the benign and malicious samples for measuring their distinguishability across different attacks, which could be calculated on real-world datasets for empirical verification.

A2. Discussions on the last-token embeddings

Thank you for the suggestion! Below, we address each point in detail:

• How does information from earlier tokens affect detection? In our approach, the last-token embedding is chosen because it aggregates contextual information from all preceding tokens during the forward pass through the transformer layers. This is a common practice in representation-based analysis [3,4], where the representation of the last token is designed to reflect the cumulative context of the input sequence, including earlier tokens. Empirically, we found that embeddings derived from earlier tokens (e.g., 1/3, 2/3 position of the input prompt) often capture incomplete contextual signals (see table below), leading to less effective separation of malicious and benign prompts. Here we report the performance on detecting adversarial meta-instruction using LLaVA.

• What properties of the last token make it particularly suitable? The last-token embedding encapsulates the model's "final understanding" of the input, influenced by all previous tokens, including cross-modal interactions in vision-language models. It often represents a culmination of the attention and transformation processes applied throughout the model's depth. Additionally, malicious prompts, especially jailbreaking prompts, often inject adversarial patterns towards the end of the input (which corresponds to the textual encoding part). The last token is thus more likely to reflect the intent and context of the complete prompt.
• How robust is this choice across different prompt structures? We follow the recommended prompt format for the LLaVA (https://huggingface.co/llava-hf/llava-v1.6-vicuna-7b-hf) and Phi-3 models (https://huggingface.co/microsoft/Phi-3-vision-128k-instruct) in our experiments. In the table below, we use the LLaVA prompt format for Phi-3 and Phi-3 prompt format for LLaVA, and report the performance on detecting adversarial meta-instruction. We observe that prompt structure can indeed affect the detection performance, and the recommended prompt format for a certain VLM in their huggingface repository is better for malicious prompt detection.

Thank you for your response and the detailed explanation of the geometric intuition behind your SVD-based method.

However, I'm curious to know more about how your method handles attacks that exploit interactions between modalities, such as those explored in "Dual-Key Multimodal Backdoors" (Walmer et al., CVPR 2022). These attacks rely on subtle, coordinated triggers across modalities, and the malicious behavior only emerges through specific cross-modal interactions.

It would be helpful to understand how the SVD approach captures these more complex cross-modal relationships. For instance, could you elaborate on whether the singular vectors are sufficient to effectively model such dependencies? Perhaps a deeper analysis of the singular vectors corresponding to the smaller singular values might reveal subtle patterns indicative of cross-modal attacks.

Alternatively, could you provide some insights based on your current experimental setup on how your method might perform against these types of attacks? Even if explicit tests against coordinated cross-modal attacks are not feasible at this stage, any reasoning or analysis based on your existing findings would be valuable.

[1] Walmer, Matthew et al. “Dual-Key Multimodal Backdoors for Visual Question Answering.” 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2021): 15354-15364.