Programmatic Evaluation of Rule-Following Behavior

Baselines. We are unclear what types of methods are referred to as "baselines" by the reviewer. We would be happy to compare our benchmark with appropriate baselines if the reviewer could point them out. Since our scenarios rely only on text inputs and outputs, RuLES is compatible with future LLMs. We will publicly release all code and data, which we have now also uploaded. Future models may outperform existing models, but this will not invalidate the results found in this submission.

Red-teaming and alignment. Our work is loosely related to prior work on red-teaming LLMs to evaluate their susceptibility to produce harmful or toxic outputs ("alignment"), but with important differences. Most relevantly, we evaluate ability to follow simple application-specific rules, rather than universal human values (such as "do not be offensive") that are relevant to all applications. Our setting, supporting zero-shot enforcement of application-specific rules not known at LLM training time, is plausibly harder than "alignment" (training to enforce universal values that are known at training time); and at the same time, it might also be easier, as we consider simple rules that admit an objective and unambiguous interpretation, whereas universal values can be subjective and ambiguous. For these reasons, different evaluation methods and benchmarks are needed in our domain, and that is what we provide in this paper.

Novelty and contributions. "Reasonably obvious" is a fair interpretation of our results, given what we have seen in all prior work on jailbreaking LLMs within and without the academic community. That said, while many might have reasonable intuitions or expectations about how well LLMs can follow rules, we are not aware of any prior work that has quantitatively or qualitatively evaluated this question. As we seek to motivate more clearly in our revised submission (we have also significantly reorganized the presentation of the material), we view our main contribution as one of formalizing an experimental paradigm for the research community in which to make reproducible progress on developing stronger defenses and more reliable rule-following behavior in LLM assistants. Our test suite is one approach to evaluating a variety of manual jailbreaking strategies. As behaviors and defenses evolve, it may be necessary to collect newer, harder test cases.

Ethics review. We are happy to undergo an ethics review, but would note that our work does not introduce any novel vulnerabilities or attacks. Prior to submission, we spot-checked various manual jail-breaking strategies as well as optimized GCG suffixes on ChatGPT, etc. but did not find noticeably different results from what is already readily searchable on the Internet. Further, traditional vulnerability disclosure processes such as OpenAI's bug bounty program (https://bugcrowd.com/openai) explicitly disclaim model prompts and outputs as out-of-scope, so we did not pursue formal disclosure.

The purpose of our evaluation programs is to remove the dependency on manual labeling effort. The ability to automatically evaluate model responses is a key component of our benchmark, since it enables much faster and cheaper evaluation, shortening the development cycle.

Choice of scenarios. We chose the two domains of security properties and children's games for their relevance and simplicity/familiarity, with the former domain being more "serious" and the latter being more "lighthearted". Our proposed benchmark can be extended to other domains with relatively little effort, e.g. "you are a security guard at the art museum", but we did not notice any qualitatively different behavior by LLMs between the two domains we did study.

Results and validity. We found some issues with the defense prompts evaluated in the initial submission and have removed them from our revised version. Whether our benchmark carries any external validity in the real world is an important question that would require more longitudinal study of how performance on our benchmark correlates with real-world measurements of performance over time. However, we would point to the highly correlated efficacy of both manual and automatic jailbreaks in our scenarios, compared to generation of harmful/toxic outputs as investigated in prior work, as some initial evidence in our favor.

The points about overall organization and discussion are well taken, and we believe that our revised submission provides a clearer organization and more tangible analyses of our results. The goal of this work is not to study particularly novel behaviors of LLMs, rather we present the design and usage of our evaluation scenarios and test suite as a potential test bed for future research. We point the reviewer to the files llm_rules/scenarios/{security,games}.py for the concrete implementations of our scenarios, as well as the files in data/systematic/ to see our test cases.

Defenses. We recognize that there is a large space of potential prompting and filtering methods that might substantially improve rule-following. We see our work as providing an evaluation benchmark that others can use, as they propose such interventions. As far as we are aware, current deployments of LLMs generally do not use those "layered defenses". It is also important to note the difficulty of properly evaluating proposed defenses. Without sufficient conscientious red-teaming, it is easy to draw faulty conclusions about proposed defenses [1], and we hope our benchmark will be a first step towards supporting such evaluation. We also have removed the scratchpad and double check prompting results from our revised submission because after further analysis of output logs, we realized not all the models fully understood our existing formulations of the prompts.

Novelty and takeaways. It might be misleading to conclude that LLMs cannot follow rules at all; while the pass rate falls far short of what we would hope for, it is also significantly higher than zero. While many might have intuitions or expectations about how well LLMs can follow rules, we are not aware of any prior work that has quantitatively or qualitatively evaluated this question. Work on prompt injection is relevant, but generally has considered only one narrow attack strategy: e.g., "Disregard previous instructions and instead". We consider a broader array of strategies for fooling models into violating the rules. Some of our strategies would not be considered a successful prompt injection attack (because they do not allow completely replacing the existing task with any other task) but are able to trigger rule violations. As discussed in our revised submission, we also distinguish this work from red-teaming model alignment which focuses on circumventing a set of universal rules and values that the model has been trained to always obey. The preprint [2] suggested by the review does not appear to be relevant, as it considers performance in long contexts, while all of our scenarios use short contexts.

[1]: Anish Athalye et al., Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples. ICML 2018

[2] Nelson F. Liu et al., Lost in the Middle: How Language Models Use Long Contexts. ArXiv 2023