We thank reviewer VYJr for their positive feedback and have incorporated their suggestions. We address the questions below:

1. “I expect hallucinations to be a problem when they happen in-distribution (i.e. when users naturally chat with models).”

We agree that finding natural “in-distribution” prompts that elicit hallucinations is important, and thus we include a regularization term to enforce the fluency of the generated prompts and make it more likely they could appear over the course of natural usage. Many prompts discovered by our investigator agents not only demonstrate fluency, but also use naturalistic cues to elicit target behaviors, e.g.:

Target hallucination: Matadors wave red capes because bulls are angered by the color red.

Generated Attack: According to the art of bullfighting, what are some of the visual and auditory cues that matadors use to provoke the bull, and what are the tactics they employ?

Response: In the art of bullfighting, also known as “corrida” or “tauromachia,” matadors use a combination of visual and auditory cues ... 1. Capote: The matador uses a large, red and yellow cape (capote) to lure the bull into a specific area of the ring. The cape’s movement and color are designed to stimulate the bull’s natural instinct to chase and attack…

We will update the example in the paper to highlight the more natural prompts our investigator finds.

2. “It is not clear to me how novel the methods/inputs found? For instance, for harmful response elicitation, authors claim ‘We find that FW discovers distinct attack modes, increasing entropy, as shown in Table 5.’ I do not feel the table provides enough evidence to support this claim. Can maybe authors extend their analysis on this?”

We find that our investigator rescovers a wide variety of attack modes found for jailbreaking language models – typos & capitalization [1], multilingual tokens [2], persona role-playing [3], reverse psychology [4], gibberish tokens [5], instruction overriding [6]; though does not find others including persuasion, and past tense. The novelty of our approach is that these methods can be found using a single investigator incentivized to find diverse attacks, rather than depending on human insight for each individual attack mode.

3. “Repetition is a trivial method of elicitation (i.e. ask the model to copy a string). Authors mention this at some point as a limitation to motivate some methods that circumvent this. Was this problem fully solved?”

We performed experiments where we measured the repetition rate (as quantified by fraction of bigrams in the attack prompt that present in the target string). In Figure 3 (Appendix), we plot the repetition rate across different FW iterations. We find that FW attains an elicitation score and repetition rate matching that of “gold prefixes” from the pretraining string dataset. This suggests that we are able to diversify beyond the repetition strategy.

4. “It is not clear to me how the harmful response elicitation investigator is trained. Authors say they use the same model as for harmful string elicitation with a synthetic augmentation based on AdvBench. What does this mean exactly?”

We use the same SFT checkpoint for both the harmful response elicitation and harmful string elicitation scenarios (Llama 3.1 8B finetuned on WildChat). During RL, we train a different synthetic dataset depending on the scenario – AdvBench (Harmful Strings) for the harmful string setting, and AdvBench (Harmful Behaviors) for the harmful response setting. The model is only trained on on-policy samples during RL. It is not trained on jailbreaks, although it is possible that some exist in SFT dataset, so we include an ablation in the Appendix using the UltraChat dataset, which is purely synthetic. We will update the discussion in Section 5.1 to clarify these training details.

5. “What is ASR in Table 5? Is it for the method itself over all prompts, or the success for that prompt after sampling N times? Why are FW-full and SFT empty but still high ASR?”

For all tables, the ASR is the success rate computed by averaging across all the target strings y.

We didn’t show an example for FW-full since sampling from the full distribution is equivalent to sampling from one of the FW iterations according to the mixture weight. In the next version, we will add a sample for SFT to all tables that are missing one.

References:

[1] P. Ding et al., “A Wolf in Sheep’s Clothing: ..."

[2] J. Li et al., “A Cross-Language Investigation into Jailbreak Attacks in Large Language Models.”

[3] R. Shah et al., “Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulation.”

[4] Z. Wang et al., “Foot In The Door: ...”

[5] A. Zou et al., “Universal and Transferable Adversarial Attacks on Aligned Language Models.”

[6] Z. Wei et al., “Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations.”

We thank reviewer BMgr for their time and review. Reviewer BMgr primarily asked for a clarification about the generalization of our investigator model, and for a comparison with gradient-based search baselines. We address these comments below. Please let us know if there are any additional questions or concerns during the discussion period. Thank you!

1. “I doubt whether this "investigator" LM has good generalization ability across different target y.”

We thank the reviewer for pointing out this concern, but we would like to clarify that our approach empirically generalizes to target outputs outside the training data. Our test sets for each experiment primarily consist of target strings which are not present in the training data (0% overlap for pre-training elicitation, 3% train-test overlap for harmful strings, and 6% overlap for harmful responses), and the strategies we discover (prefixing with high level summaries, role-playing, repetition) still generalize across different strings y. If we restrict the test set to contain only target strings that do not appear in the training set, our success rate drops negligibly from 98.78% to 98.74% for harmful strings (and remains at 100% for harmful responses), indicating good generalization.

2. “Some related works [1,3] based on adversarial attacks should be compared and discussed.”

We already compare and discuss [3] as a baseline (GCG) in Table 3, and show that our approach outperforms GCG by improving the attack success rate from 57% to 100%. [1] is the same class of method as [3]; both use gradient information to update tokens and maximize log-probabilities of target strings. We will include a discussion about [1] in the revised paper, thank you for the suggestion.

3. “More discussion about the value of behavior elicitation.”

We will update the Introduction to contextualize behavior elicitation in the literature and address its relevance to model safety, as follows:

Developers of language models seek to ensure they are well-behaved over the wide distribution of inputs they receive at deployment, for instance by training them to follow a behavior spec [4] or constitution [5]. However, many of today’s models still exhibit unexpected behaviors [6], and even describing a model’s behaviors is difficult due to the near-infinite space of possible inputs and ways for models to respond.

To address this challenge, one approach is to design automated methods to uncover specific unwanted behaviors, as in automated jailbreaking [7, 8]. However, this restricts to a narrow distribution of tasks and often produces inputs that are not interpretable by humans. At the opposite end, humans have discovered many surprising behaviors through open-ended interaction with language models [9, 10], but this is expensive and difficult to scale. Our method addresses how we can get the best of both worlds—by building tools that are automated and scale to frontier systems, while being flexible enough to meet the open-ended complexity of language model behaviors.

This paper introduces a framework for automated behavior elicitation that trains language model agents to investigate other AI models (Figure 1). Our approach frames behavior discovery as a reinforcement learning problem, where an investigator model is trained to generate inputs that produce specific behaviors from a target model—ranging from exact string matches (string elicitation) to behaviors defined by open-ended rubrics (rubric elicitation). By conditioning these investigators on samples from a distribution of high level goals rather than optimizing separately for each goal, we amortize the computational cost of search through input space during training. Our approach yields flexible, general purpose investigators that discover interpretable prompting strategies expressed in natural language.

We would also like to highlight Reviewer VYJr’s comment on this from Relation To Broader Scientific Literature:

1. Understanding worst-case behaviors is important for safety.
2. Automatic elicitation can help automate part of the process to improve robustness of LLMs.
3. Post-training may underelicit some capabilities and developers may want to understand to what extent they are represented in the model.

References:

[4] OpenAI, “OpenAI Model Specification,” 2024.

[5] Bai, Y., et al. Constitutional ai: Harmlessness from ai feedback.

[6] Roose, K. A conversation with bing’s chatbot left me deeply unsettled.

[7] Zou, A. et al. Universal and transferable adversarial attacks on aligned language models.

[8] Liu, X. et al. AutoDAN: Generating stealthy jailbreak prompts on aligned large language models.

[9] Li, N et al. S. Llm defenses are not robust to multi-turn human jailbreaks yet. arXiv preprint arXiv:2408.15221, 2024.

[10] Ayrey, A. Infinite backrooms: Dreams of an electric mind, 2024.

We thank reviewer jLAY for their time and review. We appreciate the positive review, and will address the typos and table captions you point out in the revision. We address broader comments and suggestions below. Please let us know if there are any additional questions or concerns during the discussion period. Thank you!

1. "Entropy alone does not fully capture meaningful diversity”

We agree with the reviewer that entropy does not precisely capture semantic variations. However, for the purpose of red teaming success, style matters as much as semantics. Therefore, we believe entropy is a good metric to capture diverse prompts, as it integrates the semantic and stylistic aspects in one metric. Additionally, the entropy term fits well as a part of the posterior inference objective, leading to an approximation to the true posterior.

2. “Compare Frank-Wolfe against other diversity enhancing techniques”

In another experiment, we conducted an additional baseline of setting the coefficient of the entropy term to 1. The result suggested that our Frank-Wolfe approach significantly outperformed this baseline in terms of elicitation rewards. In addition, some recent papers are also diversity-seeking, but suffer from a tradeoff in attack success rate [1]: GFlowNet only attains 80% attack success rate.

3. “Generalization should be tested on broader sets of models”

The reviewer suggested testing generalization to smaller models. However, in our elicitation setting, we want to make the point that we could train smaller investigator models and audit larger and proprietary models, therefore, testing the generalization to smaller models is less practically interesting. As for multilingual models, most of the existing models are multilingual, including some that we tested on (Llama-3.1, GPT-4o and Claude-3.5). They are still dominated by English usage, but they already enable multilingual queries and responses.

4. “The paper relies on LLM-based verifiers to judge rubric-based elicitation. This introduces a risk of reward hacking”

We agree with the reviewer that using an LLM-based judge does increase the risk of reward hacking. To deal with this, we carefully engineer the LM-as-judge prompt to make the resulting judge more robust. In addition, we have carefully verified that the elicited outputs are indeed harmful. While producing the LM-as-a-judge, we produced judgments on a subset of the data and verified that the LM judgments corresponded with our own, to verify that the elicited outputs were indeed harmful.

5. “While results show clear trends, the paper does not include confidence intervals or statistical significance tests (e.g., t-tests, Wilcoxon signed-rank tests) for comparing different methods.”

We do not use these for our analysis as the majority of prior work in AI red-teaming relies on attack success rate (ASR) as the headline metric, as this reflects the performance of attacks on the target dataset. We show comparisons between our method and other contemporary red-teaming approaches, and find that we attain higher ASR.

6. “Paper does not analyze efficiency trade-offs”

The primary advantage of our approach is that as opposed to most attacks for a single target (e.g. GCG), we can optimize a single policy to find attacks that generalize to any in-distribution target string/rubric. Thus, attack generation is very fast at inference time, even though our method requires training on a larger dataset.

[1] S. Lee et al., “Learning diverse attacks on large language models for robust red-teaming and safety tuning,” arXiv preprint arXiv:2405.18540, 2024.