LlavaGuard: An Open VLM-based Framework for Safeguarding Vision Datasets and Models

Thanks for your detailed response and the constructive feedback! Below we address your concerns.

---

1. Generalization Beyond the Held-out Test Set

While achieving strong performance on the held-out test set is not a small step, we agree that exploring additional datasets provides valuable insights. For this reason, we evaluated LlavaGuard on several real-world (e.g., ImageNet) and synthetic datasets (e.g., Stylebreeder), with users verifying LlavaGuard assessments (see overall score Tab. 3). Below, we break down those scores across datasets. These user evaluations demonstrate that our approach generalizes well beyond the original training and test sets. While individual dataset scores should be interpreted with caution due to low support (e.g., some have only 40 examples annotated), the overall generalization of LlavaGuard remains well-supported.

We have not identified any suitable benchmarks with safety annotations, as most existing ones primarily focus on (conversational) model responses rather than analyzing image content directly (e.g., MSTS).

2. Clarifications on PER/PES Metric

As explained in Sections 4 (lines 182 ff) and 5 (lines 246-252), we use a combination of PER and balanced accuracy to ensure metric robustness to data imbalance. In a dataset with a dominant class, e.g. "unsafe" samples, there will be more policy exceptions labeled as "safe". A safety classifier that always predicts "safe" will naturally classify many policy exceptions correctly, inflating its PER. However, this can be misleading, as it may give the illusion of greater flexibility than the model actually possesses. To account for this imbalance, we introduce PES, which combines PER with balanced accuracy. PES can be seen as a more reliable measure for policy modifications. That said, we recognize the standalone value of PER (see table below, Response3) and included it in the paper.

3. Comparison with other VLMs

Recently, there has been a surge in strong VLMs, and we have now evaluated several models: Qwen2.5, InternVL2.5, and the reasoning model QvQ. All models initially showed lower performance, with Qwen models scoring 67-71%. Moreover, we developed QwenGuard using our framework, once again outperforming their baseline models by a significant margin (bal. acc. of 88-90% and PES of 85%, see Table below), on par with LlavaGuard.

We also investigated building SiglibGuard (based on Siglip2-large) as a lightweight alternative to LlavaGuard. Notably, the Siglib model does not natively support safety classification, meaning there is no direct baseline. Instead, to adapt it for safety evaluation, we train a classification head on top. While SiglibGuard outperforms all VLM baselines, it still lags behind LlavaGuard and other VLM-Guards by more than 16%. A further central limitation is that CLIP-like models cannot inherently process (varying) safety policies. The safety policy provided as input to the VLMs offers a strong and beneficial inductive bias, which CLIP-like methods are missing.

When examining reasoning models, QvQ reaches only 60% balanced accuracy. In analyzing QvQ's failure cases, we observed that it often got lost in recursive reasoning traces, highlighting the complexities of safety evaluation--a promising area for future research. For example, our generated rationales could serve as training data to develop safeguards with safety reasoning traces.

4. Related Works on CoT and Visual Reasoning

We agree, and this citation ("Chain-of-Thought Prompting Elicits [...]") was actually already included in our bibliography, but accidentally missed including it in the main text. We will revise our related work section about visual CoT and reasoning accordingly and add further relevant references, such as "Measuring and Improving Chain-of-Thought Reasoning in Vision-Language Models", focusing on consistency in visual reasoning.

5. Typographical Errors and Figure Clarifications

Thanks, fixed it!

---

Thanks for considering our responses. Please also see our responses for the other reviewers. Given our responses, we would appreciate it if the reviewer reconsiders their score.

Thanks for your detailed response and the constructive feedback! Below we address your concerns.

---

W1 Additional VLMs

According to your suggestion, we included additional prominent VLMs (e.g., Qwen-VL) and baseline image-centric models (SigLip2), please refer to Response3 for LJnY.

W2,W3,Q1 Future strategies to adapt LlavaGuard to real-world safety rules

While LlavaGuard already exhibits preliminary adaptability (e.g., through our policy-driven inference), we acknowledge that customizing it to evolving, real-world compliance regulations across diverse domains presents important future challenges. To effectively enable more complex policy changes without requiring resource-intensive retraining, we envision the following strategies:

1. Rule-Based Classification. Modular, rule-based classifiers that operate on top of the VLM’s intermediate outputs--such as structured rationales or even extracted semantic symbols from rationales. These symbols can serve as inputs to a logic-based reasoning layer (e.g., a logic program), which can be updated based on evolving safety taxonomies.
2. Inference-Time Reasoning. Novel scaling methodologies beyond model parameters can offer significant benefits for adaptability. For instance, scaling inference-time compute to incorporate step-by-step reasoning traces can enhance nuanced safety understanding and, in turn, allow more nuanced adaptations of safety specifications.

W4 Subjective Rule Definitions.

This is indeed a crucial concern. Subjectivity of (safety) taxonomies is a well-known challenge, not only in recent AI safety taxonomies but also in other real-world applications and guidelines, such as PEGI, or similar frameworks, where classification rules require subjective interpretation. For example, PEGI’s treatment of simulated gambling recently led to a controversial age rating that was later successfully appealed (see IGN article), highlighting challenges of subjectivity in the realization of such taxonomies.

A key motivation for leveraging VLMs like LlavaGuard is their capability to handle both precisely defined and inherently subjective or vague safety guidelines through their acquired commonsense understanding. However, we recognize that subjectivity remains a challenge—LlavaGuard likely reflects some averaged subjectivity embedded in its training data rather than an objective standard.

Though we did not directly analyze LlavaGuard’s handling of subjective guidelines, results from our user study demonstrate strong human-model agreement—including subjective and ambiguous cases—indicating that LlavaGuard’s assessments generally align with human interpretations.

However, we acknowledge that biases embedded within the model’s acquired commonsense understanding might influence specific subjective decisions. We agree that systematically investigating, understanding, and mitigating these biases is an essential direction for future research. We will add this to our discussion.

Q2 Extension to Video

Thank you for highlighting this important direction. Technically, LlavaGuard could be adapted to video scenarios using a sliding-window approach, processing videos frame-by-frame or through short segments, and aggregating frame-level safety assessments into an overall video compliance score.

However, videos inherently combine multiple modalities, most notably audio alongside visual content. Therefore, a more robust approach would involve extending our current vision-language models (VLMs) to multimodal architectures capable of jointly modeling visual, textual, and auditory signals.

A particularly relevant and impactful application could involve compliance verification for video games, films, and streaming content, which currently rely on established rating frameworks such as PEGI or ESRB. Our vision includes developing multimodal safeguards capable of automating and augmenting traditional human-based rating processes, ensuring consistent and transparent safety assessments at scale. Additionally, with recent advancements in generative AI, we anticipate the emergence of interactive applications where generative models dynamically create video clips in real time, based on user prompts or interactions. In such applications, real-time safety checks become crucial, as the dynamic nature of generated content could rapidly introduce risks that traditional compliance approaches might not detect. LlavaGuard, due to its robust multimodal understanding and flexible policy adherence, and follow-up advancements, could play an essential role in monitoring, assessing, and ensuring the safety of these generative interactive media environments.

---

Thanks for considering our responses. Please also see our responses for the other reviewers. Given our responses, we would appreciate it if the reviewer reconsiders their score.

Thanks for your detailed response and the constructive feedback! Below we address your concerns.

---

1. Annotators and Dataset Information

We reaffirm our commitment to ethical and regulatory standards. To ensure annotators' well-being, dataset annotation was directly performed by the authors, and our user studies included researchers with expertise in handling safety-related content, following annotator protection guidelines by Vidgen et al. (2019). For more details, please refer to App. J.

Regarding the dataset, all annotators are male, White, and between 20-40 years old. We adopted a prescriptive annotation approach, collaboratively developing a taxonomy that defines a detailed categorization. For edge cases, all annotators discussed potential contradictions to achieve consensus. For the user study, participants were aged 20-30, 30% male and 70% female, with 70% identifying as White (from the US and Europe) and 30% as Asian-American. Despite the demographic differences between dataset annotators and user study participants, we observed a high level of agreement.

Regarding copyright compliance, we adhere to Fair Use principles. Efforts have been made to ensure that images are in the public domain or under open licensing. Most of our data is based on SMID (Creative Commons license). For the remaining data we collected, neither the images are distributed, nor does the model distribute data. Consequently, training our model on this data does not infringe copyright, as it serves a non-commercial research purpose.

We will incorporate these clarifications to enhance transparency.

2. Classifier Baselines

We evaluated simpler approaches like CLIP-based classifiers (Q16, App. Tab. 5), achieving significantly lower performance (only 69% acc.). We have also finetuned SigLIP2-large, which reached 99.6% train and 73.7% test acc. (more details in Response3 to LJnY). While the performance is higher than Q16's, it still lags behind LlavaGuard's accuracy by more than 16%. The main reason is that CLIP-like models cannot handle (varying) policies. Testing SigLIP under fixed-policy conditions yielded improved acc. (79%), still far below LlavaGuard.

3. Rationale Quality

We acknowledge that a quality validation of guided rationales strengthens our findings. Therefore, we present a more comprehensive evaluation using GPT-4o to compare guided vs. non-guided rationales across our entire dataset. The GPT-4o judge scored rationales based on comprehensiveness, accuracy, and guideline adherence:

The results demonstrate that our guided generation approach produces substantially better rationales, which is supported by qualitative examples in App. Fig. 7. Additionally, we benchmarked rationale quality across Llava model scales:

We used Llava-34B in our final setup to generate rationales for LlavaGuard training and the table shows it largely outperforms the other base models. These clarifications have been added to the paper.

4. Additional VLMs

We fully agree about the need for a broader evaluation range, particularly examining strong recent models (e.g., Qwen-VL series). Please see our detailed Response3 to LJnY.

---

5. Failure Case Analysis

Upon examination of false-positives, we noted common failure cases of LlavaGuard:

• Images near decision boundaries proved inherently challenging.
• The model occasionally outlined challenges interpreting embedded image text. For instance, it misclassified signage explicitly prohibiting sexual harassment of nurses as unsafe content.

We uploaded an overview (Please note some samples might be disturbing). We add these insights to our paper.

6. Downstream Performance on Filtered Dataset

We trained ResNet50 (50 epochs, LR=0.001, AdamW optimizer) from scratch on original vs. LlavaGuard-filtered ImageNet (~20K images removed, ~1% overall).

Despite removing up to 55% of samples in some classes (e.g., “assault rifle,” “army tank,” “missile,” “syringe”), overall downstream performance remains unchanged. Inspecting the most filtered classes only, we observe a gap.

This shows that careful safety filtering does not have to come with compromises in downstream performance.

---

Thanks for considering our responses. Given our response, we would appreciate it if the reviewer reconsiders their score.