JAILJUDGE: A Comprehensive Jailbreak Judge Benchmark with Multi-Agent Enhanced Explanation Evaluation Framework

This paper proposes a comprehensive jailbreak judge benchmark with a multi-agent enhanced explanation evaluation framework, motivated by the challenge of existing evaluation methods lacking explainability and generalization to complex scenarios. To address the previous issues, the JAILJUDGE dataset comprises over 35k+ instruction-tune training data with reasoning explainability and test data with 4.5k+ broad risk scenarios and 6k+ multilingual scenarios. To provide reasoning explanations and fine-grained evaluations, this work also proposes a multi-agent jailbreak judge framework to enhance the evaluation quality with explicit interpretation. With the instruction-tuned model, an attacker-agnostic enhancer and a safety moderation defense method are designed based on the reliable evaluation benchmark. Comprehensive experiments are presented to evaluate the effectiveness of the proposed methods.

The paper proposes a dataset, a judge model, and a defense model meant to study unsafe responses by LLMs.

This paper presents JAILJUDGE, a comprehensive benchmark and evaluation framework designed to assess jailbreak vulnerabilities and defenses in LLMs, which includes a wide range of risk scenarios with complex malicious prompts (e.g., synthetic, adversarial, in-the-wild, and multi-language scenarios, etc.) along with high-quality human-annotated test datasets. To enhance interpretability, the authors propose a multi-agent judge framework, JailJudge MultiAgent, which evaluates LLM responses with fine-grained scoring (1 to 10) and provides reasoning explanations. Additionally, JAILJUDGE Guard, an end-to-end jailbreak judge model, is introduced to offer explainable evaluations without incurring API costs. The paper also demonstrates two applications: JailBoost, an attack enhancer that increases jailbreak success rates, and GuardShield, a defense mechanism that significantly improves LLM robustness against jailbreak attempts.

This paper introduces JAILJUDGE, a comprehensive evaluation benchmark, and framework for assessing LLM-as-a-judge for safety settings. The authors create two main components: (1) A large benchmark dataset (JAILJUDGE) containing 35k+ training examples and 10k+ test examples spanning multiple languages and attack types, and (2) A multi-agent framework for evaluating jailbreak attempts that provides explanations and fine-grained scoring. The authors also develop JAILJUDGE Guard, an open-source model trained on their dataset that can evaluate jailbreak attempts. Experiments show their approach outperforms existing methods across multiple metrics.

This paper proposes a comprehensive jailbreak judge benchmark with a multi-agent enhanced explanation evaluation framework, motivated by the challenge of existing evaluation methods lacking explainability and generalization to complex scenarios. To address the previous issues, the JAILJUDGE dataset comprises over 35k+ instruction-tune training data with reasoning explainability and test data with 4.5k+ broad risk scenarios and 6k+ multilingual scenarios. To provide reasoning explanations and fine-grained evaluations, this work also proposes a multi-agent jailbreak judge framework to enhance the evaluation quality with explicit interpretation. With the instruction-tuned model, an attacker-agnostic enhancer and a safety moderation defense method are designed based on the reliable evaluation benchmark. Comprehensive experiments are presented to evaluate the effectiveness of the proposed methods. However, there are several points to be further improved. For example, the methodology or mechanism innovation is limited although the primary area of this work is dataset and benchmarks. In general, the difference between this work from the previous jailbreak judge benchmark or methods is not significant according to the summarization of Table 1. Specifically, compared with WildGuard, JAILJUDGE incorporates multilingual languages; while at the method aspect, the differences are mainly on the open source model and data. The claimed contributions are overwhelming but not clearly compared with the previous research works in each aspect (e.g., the jailbreak evaluation benchmark, the jailbreak attack, the jailbreak defense, and the explanation ability of LLM), which makes it hard for readers or reviewers to find a unique fundamental innovation of the benchmark with the proposed attack and defense methods although they are comprehensive. Moreover, there are concerns about the quality of the underlying dataset of vanilla prompts that can be addressed by more detailed discussions of quality metrics for the human annotation process and expert relabeling of conversations. Manual inspection of the anonymized dataset reveals that several prompts are not complete sentences and it is not clear that many of them would truly violate the MLCommons line. For example, some prompts solicit instructions on building a gambling website. Besides, the organization and presentation of the current version are not good, although the reviewer can be impressed by various contributions to the direction of research. It could be better if the author could significantly improve the presentation with clear problem illustration and state the rationality of the proposed evaluation framework. Therefore, this paper cannot be accepted at ICLR this time, but the enhanced version is highly encouraged to submit other top-tier venues.

Reject