Reducing the Scope of Language Models with Circuit Breakers

While I find the paper very interesting, there are multiple concerns regarding the results and presentation.

The quality of the rejection detectors is not evaluated

The different methods used seem to rely on training heuristic detectors, like keyword-matching or repetitions. How well do these detectors perform in practice? It seems they were trained on very small samples of 30 examples, but how much of the model rejection/acceptance behaviors do they capture? Specifically, the authors describe cases where the model first rejects a query but then as generation continues it moves to accept it. This behavior is not explicitly measured (even qualitatively). Ideally, for scoping, we would like to see a rejection without moving to acceptance later on. I guess one could simply use a detector and based on that stop the generation, but then no evaluation of that detector was conducted. Therefore, whether a method yields a high rate of such rejection→acceptance cases and how many of those are detected is a factor that should be measured. Also, the fact that rejection is measured on a heuristic basis and with different detectors for different methods could be noisy. I’d expect to see a qualitative comparison in addition to the quantitative results, or numbers that justify the quality of these detectors on a larger sample.

Robustness of the results

I am concerned by the fact that the vast majority of experiments are limited to one model and a single system prompt (that was also used in training some of the methods).

• Regarding the model – the vast majority of experiments in the paper are performed on Mistral 7B. Results on an additional model (Granit 7B) for the robustness experiment are provided in the appendix, but seem not to agree with the results for Mistarl. Specifically, CB often does not seem to be robust on OOD samples, which has been highlighted as its main advantage over other methods. Taken together, the observations (at least for this experiment) are not conclusive and show that more evaluation is needed for this experiment.

• Regarding the prompt – This could be problematic since models can behave very differently for paraphrases and perturbations. Examples of the dramatic effect of this on evaluation are https://arxiv.org/abs/2401.00595 and https://arxiv.org/abs/2310.10062.

While I understand the authors’ argument that all tested methods have been evaluated on multiple models, their tradeoffs and performance on scoping were not tested. If the paper argues for a comprehensive evaluation it should be more careful about these choices. Generally, I think the fact that the results are not conclusive for Mistral and Granit is fine as long as the paper is framed as an evaluation paper. But then it is important to understand those differences, perhaps by extending the robustness evaluation to another model like LLaMA (with different specs, like size and training data). If the experiments are expensive to run, then it would be valuable to provide at least some evidence that reduces these concerns. For example, an analysis that demonstrates on 1-2 methods that the prompt does not affect the robustness results substantially.

Readability and presentation

a) Since the paper poses the problem of scoping, I would expect section 4 (results) to start by answering this question, namely, how well do methods perform on scoping? The experiments cover many axes but this question remains unaddressed, until realizing there are some results for this in Appendix A.2 (but it is not referenced at the beginning of section 4).

b) The title makes it sound like this is a paper that proposes a method rather than an extensive evaluation. As such, the reader could expect a more fine-grained analysis of CB performance. I believe the title could be more informative by focusing more on the evaluation part rather than on CB as of now.

c) I found some parts of the paper hard to read, and there are many typos and some ungrammatical or unclear sentences. The main figures are difficult to digest so it is challenging to see the trends and easily understand the results. I now elaborate on all these points.

• It is hard to read Figure 1 before reading section 3, since it is not clear what the reported metrics are.
• The main results in Figures 1 and 2 are hard to parse. It’s difficult to compare the numbers because the bars are small and there is a lot of information, which makes it hard to see the general trends. Consider averaging over the different attacks and report mean and standard deviation results in a table or a simplified plot.
• The beginning of section 4 is redundant to section 3. It is also redundantly repeated many times throughout the papers that the number of experiments is large.
• Example typos:
  • Line 457: “is it possible to still reject tasks well when there are multiple tasks in the rejection set” I believe the authors mean “in the acceptance set”.
  • Line 86: typo “has many broadly performs more robustly”
  • The end of introduction “We show unlike other methods” (missing “that”)
  • Line 164: missing a period at the end of the sentence.
  • Line 521: type “it may expensive in practice”

Limited Scope of Interventions: The paper focuses solely on model weight updates (computing Δ), ignoring important external intervention techniques like input-output safeguards or LLM judges that are commonly used for task scoping in practice. See for instance the references: "NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Rails", Rebedea et al. "Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations", Inan et al.

Unrealistic Dataset Selection: The "accept" tasks (Sentiment Analysis, Summarization, Program Execution) are narrow NLP tasks that don't reflect the diverse, flexible use cases of LLMs in real applications. More realistic datasets like Alpaca are only used as out-of-distribution tests.

Evaluation Methodology Issues:

• ROUGE-L score is used inappropriately for creative tasks like story completion where multiple valid outputs exist. In practice as the tasks Sentiment Analysis, Summarization, Program Execution are the main used in the "accept set", this doesn't radically change the main experimental results. However, the figure 4 includes "Story completion", and "Dialogue generation" in the "accept" set. The choice of ROUGE-L lacks motivation, and recognition of its limit for creative tasks.
• The rejection detection system lacks validation on held-out data and uses different detection methods for different techniques (especially CB), potentially biasing results.
• In Figure 2, for the robustness evaluation under adversarial attacks, the task performance is measured under attack (this is why for instance, the task performance is always ~ 0 for the base64 attacks, because the model cannot meaningfully process request encoded in base64). ID reject and OOD Reject should used the adversarial attack, but I don't understand why use the attack to measure the accept task performance and refusal rate.
• The system prompt (Sys.) method only includes a high-level description of the tasks, and doesn't include few shot example of the 'accept' or 'reject' task. This is likely to greatly underestimate the effectiveness of prompting. Even if I expect prompting alone to be less effective than CB and DPO, I would have liked to see more effort put in the prompting baseline.

Unclear Problem Definition:

The expected behavior for out-of-distribution tasks is not explicitly justified, and the assumption that all non-accept tasks should be rejected seems artificial and potentially limiting for real applications. In particular, in the case where only one task is included in the "accept", and the "reject" task, it is unclear to my why we should expect the "right" generalisation to be "Accept only the 'accept' task, and reject every other task", and not for instance the symmetric case "Reject only the 'reject' task, and accept every other tasks".

1. Limited Data Transparency: I believe that the greatest contribution of this paper is to propose this scope-reducing task and make a solid testbed for evaluating scoping behaviors. However, the testbed and the data they used are not open-sourced, and I don’t see a commitment from the authors to make them available. This will largely decrease the contribution and the soundness of this paper.
2. Limited Coverage of Overlapping Scopes: The paper does not address cases where different scopes overlap, which can be common in real-world applications (e.g., a user query about storytelling could potentially involve the leak of privacy, like the famous grandma exploit strategy). Ignoring overlapping scopes limits the study’s relevance, as managing such intersections is crucial for robust scope control in practical deployments, instead of the narrow ones (which only involve binary judgment) provided in the paper.
3. Lack of Fine-Grained Analysis on Failure Cases: The paper does not provide an in-depth examination of cases where current methods fail, especially for adversarial prompts. A detailed analysis of failure modes would have been valuable for understanding limitations and improving future scoping methods.
4. Lack of Ablation across Model Sizes: The paper does not explore how the effectiveness of scoping methods, particularly interpretability-related methods, and prompt-engineering methods, might vary with different model sizes. Instead, the paper performs its experiments on models with similar size. Since smaller and larger models may respond differently to scoping interventions, an ablation across model sizes would provide valuable insights into the scalability and adaptability of these techniques for various deployment scenarios.

1. While the study spans many experimentations, only one relatively small LLM is covered (Mistral-7B-Instruct-v0.2). This makes assessing the generality of the overall claims relatively non-straightforward or impossible. While I am aware that running the suite of all of these experimentations is relatively computationally costly for various models at scale, having only a single model (I also saw Granite-7B used, but the results seem different for it, confirming this point) does not allow me to conclude that the presented analysis is generalisable across strictly. What happens as the model scale increases or decreases (2B,3B,13B,30B,100B, 100B+) ? Do the claims stay true? Is CB and SFT+CB as efficient with more or less parameters? Do other methods gain anything at scale?

2. The authors present a method for verifying if the output is rejected by using an early keyword lookup or sequential repetitions ("circuit broken") of the same token. While 30 samples per accept, reject, and OOD reject were chosen for tuning the threshold on these heuristics, it is not obvious how robust this combined method is given the overall diversity within the training/validation set of tasks. The amount of samples does not seem representative for such a diverse dataset. Has the method been extensively tested within SNI?

3. It is very hard to parse all of the figures within the paper due to the visibility, scale and colour schemes used. Also, as the x-y scaling of the sub-figures within the figures is different, it is very hard to assess the changes and differences in overall methods performance (i.e. accept maintained performance vs rejected in-domain and OOD changes/ratio).

4. While SFT + CB seems to be a fairly consistent (with some exceptions) leader in performance w.r.t. all of the objectives (accept performance and not-filtering vs rejected in-domain and OOD filtering) along with being robust to adversarial prompts, the much simpler method of probing is also fairly consistent when rejecting/filtering (in most of the setups) in-domain and OOD rejected samples. The difficulty that it can also over-filter the accepted set seems to be relatively overcome-able by maybe training a larger MLP (or fine-tuning a pre-trained model) or using additional heuristics along with the Probing. Also, probing seems much better than any method in "multiple accepted topics" evaluation, showing that achieving good scoping can be as simple as training a probing model. With this we might assume that Probing might be a much stronger benchmark than presented. Is there a more extensive comparison of probing techniques with SFT+CB?

Minor Corrections:

line 43: "give give" -> "give"

line 299: "to to" -> "to"

line 457: I assume the text should say "when there are multiple tasks in the accepted set"