GuardVal: Dynamic Large Language Model Jailbreak Evaluation for Comprehensive Safety Testing

• The introductory and background sections are overly detailed, occupying substantial space before introducing the method almost on page 5. I'd resume these sections to around 2 pages to allow for a more detailed explanation of the core methodology.
• The Optimizer component, crucial in the evaluation pipeline, lacks sufficient explanation. I would better clarify its role and importance within the process.
• While the reason to combine attacking and defensive scores is explained, I don't understand this approach's rationale. Additionally, as the attacking score includes generation and evaluation, differentiation of specific model capabilities becomes complex. Although presenting both attacking and defensive metrics is important, I'd keep them separated.

1. The rationale behind the overall safety value, especially the offensive capability. Although highlighted in Discussion, this reviewer is concerned about the rationale behind the design of offensive capability. Specifically, according to Equation 1, we can conclude that if an LLM's attack capability is stronger, then its OSV value will be worse. The authors should articulate a deeper rationale for such a design, e.g., why an LLM's ability to attack a target model can be correlated with its own security, and are they negatively or positively correlated?

2. The technical contribution of this paper is weak. As a benchmark paper, technical contribution is not a core evaluation indicator. However, the attack framework (like the design of role-play paradigm, including Translator, Generator, and Evaluator) has been documented in existing papers, like PAIR and TAP, and other jailbreak papers based on genetic algorithms [1, 2]. This makes this paper more like a comprehensive experiment based on these studies. The authors should clearly demonstrate the difference between this paper and existing works in terms of the attack/evaluation methods.

3. Lack of sufficient analysis of experimental results. Some results in Table 1 are counterfactual. For example, as an attacker, GPT-3.5 performs well in all LLMs, except the Vicuna. However, Vicuna is not a well-defensed LLM. Such data in the same row seems very unintuitive and the authors should provide more analysis. The same situation occurs with the use of LLaMa2 to attack Gemini.

[1] GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts. [2] OPEN SESAME! UNIVERSAL BLACK BOX JAILBREAKING OF LARGE LANGUAGE MODELS.

1. In my opinion, presentation is one of the biggest issues holding this paper back. The background and OSV sections feel too verbose, and the jailbreak generation method is barely described in the main section of the paper.
2. There are other methodological concerns as well. The paper claims that the Optimizer results in a more diverse set of jailbreak attacks, but no analysis is presented confirming this hypothesis.
3. The jailbreak generation method is not compared to other methods in recent literature, such as GCG (https://arxiv.org/abs/2307.15043) and PAIR (https://jailbreaking-llms.github.io/).
4. The paper mentions that an uncontaminated test set is crucial for safety evaluation. However, the proposed method does not guarantee this: LLMs can simply verbatim memorize jailbreaking scenarios in their training data and regurgitate them during synthesis. This is especially true for recent LLMs, as large subreddits are now dedicated to jailbreaking LLMs (r/ChatGPTJailbreak for example).
5. There are also design choices in the jailbreak generation pipeline that require more explanation. Why is the attacker LLM allowed to decide what the rejected response should be? This is standardized in most recent literature. Why is semantic similarity used in scoring jailbreaks? Harm evaluation is a complicated problem, and most automatic evaluation approaches use sophisticated prompting to determine if generated responses contain harmful content. There is also no analysis of the alignment of the Evaluator responses with human preferences, making the results less trustworthy.
6. The number of rounds required for an attacker LLM to jailbreak the defender LLM can vary substantially between test cases. This variance is not accounted for in OSV. Whenever a new LLM is introduced, OSV requires using this LLM as an attacker and defender against every other LLM in the benchmark, limiting its practicality, while also recomputing OSV numbers for every other LLM in the benchmark.
7. Some more motivation is also needed for why the capability of an LLM to jailbreak other LLMs is a factor in deciding how safe it is.

1. The calculation of Overall Safety Value seems to be a bit arbitrary. Is Offensive Capability considered as important as Defensive Capability in the formula? Will LLMs with good offensive capability get advantages that are more than expected? Is there any correlation or consistency between defensive and offensive capability of LLMs? Analyzing values and ranking with either one capability respectively may help empirically "justify the relationship between offensive and safety".