ELITE: Enhanced Language-Image Toxicity Evaluation for Safety

We greatly appreciate the reviewer’s valuable feedback, which has significantly improved our work.

Essential References Not Discussed. We agree that since our study aims to improve the safety evaluation of language models, it is important to cite DecodingTrust, an early benchmark in LLM safety. We appreciate the suggestion and will include the citation.

Other Strengths And Weaknesses. As noted in the review, the main contribution of our paper lies in improving the safety evaluation of VLMs. To this end, we propose the ELITE evaluator, a rubric-based evaluation method that provides more accurate assessments based on the characteristics of VLMs, along with the ELITE benchmark, an effective tool for VLM safety assessment. Through the analysis of existing benchmarks (Table 4) and human evaluation results (Table 6, including additional evaluations conducted in response to Reviewer BuCp’s Question 3), we demonstrate the novelty of both the evaluator and the benchmark. In particular, our work presents a simple yet effective method for evaluating harmful responses from VLMs, which we believe is a key strength of the paper. We will revise the paper to highlight these aspects better.

Question1. As VLMs interpret inputs through the interaction of two modalities, novel attack types may arise from this cross-modal reasoning process. Therefore, a promising direction is to develop comprehensive benchmarks that can broadly evaluate unsafe behaviors in VLMs and use these benchmarks to guide improvements in safety alignment. In this context, our ELITE evaluator and benchmark were designed to address these challenges, and we believe further efforts to improve safety alignment will remain essential going forward.

We truly appreciate the reviewer's insightful and constructive comments, which have greatly contributed to enhancing the quality of our work.

Methods And Evaluation Criteria & Experimental Designs Or Analyses-1. Thank you for your valuable feedback. Our key message is to address a limitation in existing evaluation methods—namely, their inability to accurately assess harmfulness in VLMs. As the reviewer pointed out, our benchmark was designed with the "difficulty" of evaluating harmful responses as a primary principle. We believe that a safety benchmark should be able to assess model robustness using sufficiently challenging samples. However, existing benchmarks often contain ambiguous samples, making it unclear whether they could induce harmful responses. Therefore, we prioritized fulfilling this fundamental role of a benchmark above all else.

Furthermore, we aim to propose a benchmark that incorporates a wide range of diversity. In particular, there are known ways to elicit harmful responses using safe image–safe text pairs, but most existing benchmarks lack coverage of such safe-safe cases. To address this, we constructed a more comprehensive benchmark by integrating a “New” dataset that includes all four types of image–text pairs.

Experimental Designs Or Analyses-2. For the analysis by features, human labeling was conducted for 30 samples using three labelers, evaluating 10 features. The table below shows the results of Lasso regression based on unsafe and safe. These results highlight the effectiveness of the ELITE evaluator's toxicity score in the VLM task.

Weakness1 & Other Comments Or Suggestions. We agree with the reviewer that human evaluation is a crucial component, especially given that we are proposing a new evaluation method. We will also revise the introduction section.

Question1. Since the filtering baseline model includes the relatively safe model Phi-3.5-Vision, and we selected only the cases where the score was above 10 in at least two of the three models (Phi-3.5-Vision, Llama-3.2-11B-Vision, Pixtral), we believe the approach is not so unfair. However, as the reviewer pointed out, we will add a note in the table to clarify that these three models were used as filtering baseline models.

Question2. We argue that “specific” and “convincing”, which have been used in previous evaluations, are orthogonal to “toxicity”. We found that responses can be highly specific or convincing without being toxic. This distinction is particularly important in the context of VLMs, where the model often provides detailed descriptions of the image, even when such responses do not align with the harmful intent of the prompt. This results in cases where the VLM appears to respond in a “convincing” or “specific”, but actually avoids engaging with the harmful intent altogether by focusing solely on the image. Such responses are frequent and posed a major challenge during our benchmark construction.

As we examined the 118 samples excluded during the ELITE benchmark filtering process, we found that 52.54% of them consisted of image-descriptive responses. This indicates that such cases occur frequently and can lead to miscalibrated scores when using StrongREJECT.

Question3. To assess human agreement and Pearson correlation, we analyzed the results of the human evaluation conducted on a total of 228 samples (Reviewer BuCp's question 3).

The agreement rate and Pearson correlation between the three human annotators are summarized in the table below. While there are some variations across categories, we generally observe a strong level of agreement overall. To mitigate individual annotator biases and ensure more reliable labeling, we applied a majority voting strategy across the three annotators’ safe/unsafe labels for each sample.

Question4. Our intention in Table 3 is not to claim that specific models are better or worse, but rather to show general trends. Regarding the ELITE evaluator, it demonstrates stronger performance compared to existing evaluation methods. As for Table 4, we believe that the difference in E-ASR between existing benchmarks and the ELITE benchmark is substantial enough that it cannot be attributed to the evaluator’s margin of error.

We greatly appreciate the reviewer’s insightful comments, which have been essential in helping us enhance and clarify our work.

Claim And Evidence. The table below shows additional experimental results for the latest model, gemma3, and the larger model, InternVL-2.5-26B. These results demonstrate that the ELITE benchmark shows consistent performance even with relatively large and latest models.

Weakness1. Thanks for your thoughtful review. It is true that we made minimal modifications compared to the StrongReject. Our goal is to create a benchmark and evaluator that works well generally, rather than being specific to a particular model or situation. What matters is not the complexity, but how many problems we can solve with simple changes. We believe that the ELITE method proposed in this paper can solve many issues.

We propose a simple yet effective method for evaluating harmful responses in VLMs. We demonstrate that our approach outperforms many existing Guard models and StrongREJECT, based solely on toxicity score requests, and through this, we create a toxic benchmark by filtering out samples that are not particularly harmful, addressing a key issue in existing benchmarks. Furthermore, we aim to propose a benchmark that incorporates a wide range of diversity by integrating existing benchmarks like SIUO, which only contains safe image + safe text, and other benchmarks, which contain unsafe image + unsafe text, and other combinations. We believe we can create an even broader benchmark by structuring the ELITE benchmark with both safe and unsafe pairs.

Ohter Commnets Or Suggestions. In Section 3.3, we explain the process of generating the new samples, ELITE benchmark (generated). In Section 3.4, we match existing benchmarks to the taxonomy in the ELITE benchmark, and by filtering both the existing benchmarks and the ELITE benchmark (generated), we ensure that only toxic cases remain. We will improve the writing in the section you pointed out to make it easier to understand. Thank you for pointing this out.

Question1. We understand the reviewer’s concern that LLMs (and VLMs) may not provide consistent responses. We measured the toxicity score of the ELITE evaluator a total of 10 times on the 228 samples (Reviewer BuCp’s question 3). The table below shows the average and standard deviation of the toxicity scores.

Additionally, the table below shows the Pearson correlation between the ELITE (GPT-4o) toxicity scores and human judgment, demonstrating that the toxicity scores are well aligned with human assessment.

The table below shows the Pearson correlation between human toxicity scores. As can be seen in the table, the correlation with the ELITE evaluator is higher than the correlation between each pair of humans. This indicates that, despite some variation in human judgements, the ELITE evaluator is more consistently reflecting the evaluations.

Question2. The 10-point threshold we used was selected based on experiments in Appendix A.2. Although Table 9 shows that a threshold of 10 is not optimal, Figures 5 and 6 confirm that even with this threshold, there are cases where the model's responses are sufficiently harmful. By including these cases, we propose a more comprehensive benchmark. Below are the five most vulnerable models for each threshold. Except for LLaVa-v1.5-7B appearing as the 5th most vulnerable model at Threshold 5 instead of Molmo-7B, the five most vulnerable models remain the same across all thresholds.

We sincerely appreciate the reviewer for the constructive feedback, which has been invaluable in helping us refine and strengthen our work.

Weakness2. To demonstrate that the ELITE benchmark is not overly tailored to the ELITE evaluator, we present results based on the previously adopted metric, Attack Success Rate (ASR), instead of the metrics (E-ASR) used in Table 4. These results suggest that the ELITE benchmark remains general and is not excessively influenced by the use of the ELITE evaluator.

Question1. Fine-tuning the evaluator model, as done with LLamaGuard, may lead to performance improvements. However, the experimental results from StrongREJECT [1] show that the rubric-based approach slightly outperforms the fine-tuned models. Based on this, we conducted our experiments using the rubric-based method, with the ultimate goal of proposing a more accurate approach for judgment, even within the rubric-based framework.

As a result, we demonstrate through thorough dormant evaluation and extensive experiments that our approach outperforms existing methods, even when using the same base model. Additionally, we believe that the effectiveness of our evaluation method is proven by the fact that ELITE (InternVL2.5), using an open-source model, outperforms evaluation models such as StrongREJECT with the more advanced model GPT-4o and LLamaGuard.

Reference:
[1] Souly, A., Lu, Q., Bowen, D., Trinh, T., Hsieh, E., Pandey, S., Abbeel, P., Svegliato, J., Emmons, S., Watkins, O., and Toyer, S. A strongREJECT for empty jailbreaks. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024

Question2. We conducted filtering and evaluation by providing images to the evaluator in all experiments. This is because some previous evaluation methods assess only the model's responses, making it impossible to judge the success of attack methods that require understanding contextual details, such as the case of suicide & self-harm in Figure 1-(c) or the examples provided by the reviewer. We will rewrite the paper by clarifying that evaluation and filtering are conducted with images included. Thank you for your valuable feedback.

Question3. We conducted the human evaluation on a total of 228 samples by randomly sampling 110 samples from the ELITE benchmark and 118 samples that were not included (i.e., filtered out). We included at least 20 samples from each taxonomy and gathered the opinions of 3 labelers per sample, with the final labeling determined by majority vote. In total, 8 labelers were recruited for this evaluation. We provided the input image, text, and model's response to perform the safety judgment. As shown in the table below, the significant difference between the included and excluded datasets demonstrates the quality of the ELITE benchmark.