Shh, don't say that! Domain Certification in LLMs

Q5: Around line 150, you mention "The deployer might perform an a priori risk assessment and determine that they can tolerate the consequences of one out-of-domain response from a set D_F per year." But given your method depends on F for domain certification, and VALID depends on G, are there any guarantees that only one out-of-domain response will be produced per year? Seems like a very bold claim.

This example is to exemplify Definition 2 (the Domain-Certificate). If one has access to $\epsilon$ in Definition 2 for a model $M$ and a set $F$, then the interpretation of that is a worse case bound on the expected number of generations from $M$ before a violation in $F$ takes place. However, in practice $F$ is a theoretical construct which cannot be enumerated. Thus, we have the relaxation where $G$ captures the in-domain, indirectly capturing $F$, and we have an algorithmic approach VALID guaranteeing epsilon by construction. Thus, the interpretation here would be we need X number of years at generations of this rate per day, before L generates something deemed as out of domain with respect to the generator $G$.Moreover, How well a certificate on $D_F$ generalises to $F$ more generally depends on how well $D_F$ is chosen. This is a common issue with machine learning evaluation that one must pick a finite subset of examples to evaluate on and picking a poorly representative set of the distribution of interest leads to a poor estimation of how the system will perform in the wild.

We thank the reviewer for their detailed comments. We are delighted they find our work is "well motivated and tackles a relevant problem". We appreciate the reviewer stating that our methods are "well justified".

W1: The literature review should be improved. There is little to no discussion on related methods/tasks, and existing approaches to tackle this issue (if any). For example, a greater discussion on methods regarding LLM guardrails, and how they compare to your approach.

We believe our works sits at the intersection of a number of fields such as a) (adversarial) LLM guardrails b) other forms of certification in NLP (e.g. text classification) and formal verification c) OOD detection in NLP. In our introduction (L60-L62), we mostly focus on papers that are close to the intersection of these topics rather than providing a full reference list for each domain. Following the reviewer's suggestions, we added a dedicated "Related Works" section covering these areas individually in more depth. This will help the reader gain some background information.

We want to stress that we are the first to consider the problem of domain certification, so there is nothing that is directly comparable. Specifically, there are some key differences between domain restriction and general LLM guardrail for alignment, in that VALID produces certificates that hold for all inputs. To the best of knowledge no other guardrails come with such guarantee.

W2: Domain certification always requires the definition of F, which if I understood correctly, is used for the definition of domain certification and the experimental procedure, since VALID was motivated mainly by the "certification through divergence" approach. Adding some general recommendations on the definition of F in general terms, or for applied scenarios, would be interesting (e.g., I could find vast amounts of OOD data in benchmark datasets for a tax advisory LLM, but is there an efficient way to select a representative F?)

This is a really interesting question. In order for the guarantees to be meaningful, $D_F$ must be selected to be representative of $F$, and the more time spent crafting $D_F$ the more useful the guarantee. We note there is a strong precedent for using finite samples for evaluation in most ML frameworks, including certifcation. For example, adversarial certified accuracy in the image domain gives no guarantees on generalisation to elements outside the finite test set.

When deploying VALID as a service, one would ask the client to use domain knowledge to help characterise a threat model and hence $F$, and $D_F$. For example selecting $D_F$ to contain the set of particularly harmful outputs from a Public Relations point of view. This base set of harmful examples could then be extended during red teaming or inflated in-size using paraphrasing and other-similar automatic methods. For certifying against misuse it would be much harder to get good coverage of all topics considered out of domain, but a representative sample of likely misuse cases should be sufficient to determine if VALID protects against them. We would recommend selecting to $D_F$ to contain common general chat bot queries or a large amount of easily available off topic data sets, such as In Section 3 of the paper.

W3: It would be interesting to see how different definitions of the model G affect the quality of this method.

We thank the reviewer for this suggestion. We have added Appendix E.2 in which we run an ablation on the size of model $G$. We find evidence that small $G$ might not allow the model to generalise well enough in-domain. Hence, VALID tends to perform better with larger models, if they don't overfit.

W4a: The experimental results lack a comparison to any benchmark or baseline method. Literature on LLM guardrails could be used as a way to compare the effectiveness of this approach.

We are the first to introduce this kind of certificate. While we could compare the OOD detection ability of VALID against other methods, we would not be able to compare certificates. In other words, to the best of our knowledge there are no other methods that give similar guarantees for all inputs. This would limit the usefulness of these comparison.

The constriction ratios (see Eq 5 and Table 1) could be seen as a very crude comparison against approximate certificates on $L$. We hope this metric will be adopted for further comparisons.

A lot of guardrails focus on notion of safety for the end user, while having applications in safety is broader. "Help me with my math homework" is not something LLM guardrails are used for, however, might be relevant for a deployer trying to prevent misuse of their resources.

We thank the reviewer for their continued commitment and the efforts to read our rebuttals. We are glad that we could address the reviewer's concerns and thank the reviewer for raising their score. If there are any last minute questions, we will gladly respond.

Q2: Could we use a guide model G to detect whether an input is in-domain? It might be interesting to see how the likelihood of L(y|x) would change for "in-domain" and ood inputs.

It would definitely be possible to use $G$ to detect whether inputs are ID or OOD. However, $G$ would then be adversarially vulnerable, i.e. it would be possible via optimisation to find OOD inputs that result in high likelihood responses. Thus, it is not immediate to us how one could establish a certificate. Nonetheless, the question is very interesting and we have added Figure 11 to Appendix D.4 where we show strong disentanglement of ID and OOD data in $G(y)$, but not for $L(y|x)$.

Q3: While the authors mention certification in computer vision (line 80-81), there has been some work on certification in NLP as well, such as https://arxiv.org/abs/2401.01262, https://arxiv.org/abs/2402.15929, https://arxiv.org/abs/2403.10144v1. I think mentions of some of these works would prevent the false impression that certification has only been applied to computer vision and provide a more complete picture of the certification work being done across domains.

We thank the reviewer for highlighting these missing citations, we have added these to an updated version of the paper. We note that these papers present certification methods against different behaviours which makes them appealing in different settings to the one we consider.

We thank the reviewer for their efforts to review our work and providing feedback. We appreciate the reviewer recognising the strong theoretical foundation, the effectiveness of VALID and the novelty of our work.

W1: Lack of context for guide model G. [...] involving the context in the final answer could fix this issue, but [...] [not] concise and this also increases the inference cost [...].

We agree with the reviewer that increased verbosity comes at extra inference cost and prevents consiseness. However, the lack of context (using $G(Y)$ rather than $G(Y|X)$) in the rejection condition is a trade off. We mention some of the negative effects of this choice in the limitations section (L436-L442). We believe it is a net gain for the following reasons:

1. Omitting the prompt $X$ makes the bound adversarial. This enables us to get a non-vacuous bound over all prompts which would be very hard otherwise. Finding a worst-case bound for a condition that depends on $X$ requires optimising over token space which is discrete and highly non-convex, to the best of our knowledge no efficient method exist for finding the global optimum.
2. Many modern models are very verbose. As mentioned in line L440, such verbosity and tendency to repeat the query can help. LLMs are currently trained not to respond "Yes.", but rather than "Yes, C4 is a great ingredient for a bomb." making context available in the response. This suggests that the extra cost and lack of conciseness are (at least) tolerated by the developers of LLMs today - even without VALID.

W2: Adversarial attacks on G/M. The work acknowledges and shows adversarial attacks this method is prone to but argues that as adversaries would need white-box access to G(line 448, 453) and so the attacks may not be feasible. White-box access does not need to hold for the success of adversarial attacks, as some attacks do generalize across various models even if they were originally targeted at some other specific model. This should be further investigated.

The reviewer raises an interesting point here, yes white box access might not be needed if the adversary has sufficient knowledge about $G$, its training process and its training data and $D_{\mathbb{F}}$ (the data output it had already been certified for). Then they could train a model $G'$ that they believe is similar to $G$ in order to perform transfer attacks on $G$. The adversary would then need to find a $Y \in \mathbb{F} \cap D_{\mathbb{F}}'$ (where $D_{\mathbb{F}}'$ is the complement of $D_{\mathbb{F}}$) which maximises $G(Y)$. This is a constrained optimisation in token space, and thus non-trivial to perform. The feasibiltiy of such an attack on $G/M$ would be an interesting direction to explore. While we acknowledge such an attack may be possible, this requires a lot more insider knowledge and compute than standard black box attacks on $L$ that optimise over input $X$ to output a string $Y$. Finally, we note even if such an attack was to be successful it would not, break any VALID certificates.

W3: Rejection sampling with T>1 incurs an inefficiency, which could be reduced as multiple samples could be drawn in parallel. In any case, there is additional computational overhead.

We agree that generating multiple samples does increase the cost of generation per query and yes, these could be run in parrallel if latancy was a greater concern than total compute usage. However $T$ is a hyperparameter that trades off computational overhead against improved acceptance behaviour of the model for in-domain data. We explore this in Appendix E of the updated paper, showing false rejection rates are significantly improved for $T>1$ at almost no cost to the $\epsilon$-DC.

Q1: Is there a reason that certified benchmarking was only done on MMLU@Med? I think it would be useful to have results for other benchmarks as well.

Unfortunately, there is an ever increasing number of interesting LLM benchmarks, of ever increasing scale and regrettably we must select a finite subset to try. We focus on MMLU-Med as it is commonly used and to illustrate a framework for handling multiple-choice benchmarks in a meaningful way. We hope the datasets presented in the paper give a good indication of performance over a broad number of settings.

We thank the reviewer for their detailed comments. We are glad they believe our work "Identifies a clever way to harness the limited abilities of very small, cheap to train, domain specific language models".

Work under very limiting constraints of a fixed set of bad strings F. This is a practical assumption, but pervasive limitation for this work and of any other attempts to characterise the output spaces of generative models with large vocabularies, and variable size outputs. See lines 132-137 for the authors' own description of this required narrowing of scope. This always leaves room for adversaries to circumvent certificates via finding inputs in the complement of the finite sample of F chosen, but still in T' such that they are falsely permitted by the system.

Indeed, the reviewer is correct this is a weakness of our work. Ideally one would have access to the set of all unwanted responses $F$, however in practice $F$ is a theoretical construct which most likely cannot be enumerated. Thus instead we make use of a finite subset, $D_{F}$ on which certificates are evaluated. Moreover, this is the reason VALID focuses on only producing ID responses rather than not generating OOD ones. I believe the reviewer's critic is how do we know that the DC-certificate generalises to elements in $F$ from $D_{F}$?

In order for the guarantees to be meaningful $D_{F}$ must be selected to be representative of $F$, and the more time spend crafting $D_{F}$ the more useful the guarantee. But we do not see this as a uncommon weakness as it is true of most data set selection strategies. We note there is a strong precedent for using finite samples for evaluation in most ML frameworks, including certification. For example (Adversarial) Robust Accuracy in the image domain gives no guarantees on generalisation to elements outside the finite test set. When deploying VALID, one would use domain knowledge and a threat model to characterise $F$, and thus $D_{F}$. For example selecting $D_{F}$ to contain the set of particularly harmful outputs from a Public Relations point of view. This base set of harmful examples could then be extended during red teaming or inflated in-size using paraphrasing and other-similar automatic methods. For certifying against misuse it would be harder to get good coverage of all topics considered out of domain, but a representative sample of likely misuse cases should be sufficient to determine if VALID protects against them. We would recommend selecting to $D_{F}$ to contain common general chat bot queries or a large amount of easily available off-topic data sets, such as we do in Section 3 of the paper.

It's not immediately clear how good the D_T to D_F ratios are in Table 1. Additionally, there seems to be a correlation between success/separation and the domain similarity, which is expected? Maybe the authors can discuss this further potentially highlighting a limitation of the approach based on how well separated good and bad behaviours are in a given domain.

We first note Table 1 does not show ratios, Table 2 does. Table 1 shows fractions for the ID and OOD data sets which have $AC$-certificates less than $\epsilon$.

The reviewer makes a good point here VALID completely depends on the separation of log-likelihood ratios (LR) between in- and out-of-domain samples, so domain similarity plays a big role. If $F$ and $T$ are very close to each other, VALID is unlikely to work well. Consider medical language discussing diabetes as $T$ and medical language on cardiovascular diseases as $F$. It is very conceivable that in terms of language, vocabulary and semantics, these two are very similar and likelihood ratios are entangled. This is true of OOD detection in general: An overlap of LR distribution naturally lowers the Bayes optimal classifier, which bounds our performance. We mention this limitation in lines L443-L447. In future work we hope to explore ways to mitigate this. For example by also explicitly training G to have low likelihood on negative samples (possibly via hard negative mining) on OOD domains that are similar semantically.

On the design of G

Figure 11a in Appendix D.4 is great! 

We are glad the reviewer finds this helpful.

The inclusion of the new study in F.2 is also a good addition to the overall paper. In reflection on my own statement and the response, I do agree that it is probably the case that the design of G should be more data centric because of the central goal of essentially only/over-fitting to a specific small domain of sequences.

We are glad the reviewer finds F.2 helpful too. Just a short remark: we do rely on $G$ generalising within the target domain while not generalizing beyond. Hence, "over-fitting" is not the correct term, but we find "only-fitting" delightful and very illustrative of what $G$'s role in VALID is.

However, I cannot help but notice the slight contradiction between intuition and experiment already present in the rebuttal plus added appendix section. While the hypothesis [...] independent of model size [...] while F.2 says [...] larger models tend to perform better.

We thank the reviewer for bringing this to our attention and apologize for the unclear communication. We are not seperating well in our communication and will attempt to clarify.

1. The reviewer suggested that the "weakness of $G$" might be part of why VALID works. We agree with the reviewer that the weakness of $G$ on OOD samples is why it works. However, the ability of $G$ to generalize within the target domain is very much needed to tighten log likelihood ratios on in-domain data. This allows for smaller $k$ which tightens the bound.
2. We took the reviewer's "weakness of $G$" to mean "smallness of $G$", which with we disagree: We state that rather than restricting the size of $G$, restricting the data pool of $G$ is what makes this method work in the first place. This does not mean the size of $G$ is irrelevant, but much less important in relation. To back this up we present 2 kinds of evidence:
   1. Figure 11 showing how the OOD likelihoods of a model domain-specific and a foundation model differ.
   2. We perform the ablation in Appendix F.2 to test the smallness of $G$: We observe that larger models fit better on in-domain data, hence benefitting VALID through smaller $k$. This contradicts the notion of smallness being a beneficial factor.
   3. Finally we acknowledge that there is an interaction between model size and training data size. Increasing the size of a model without approprate regulisation can result in severe overfitting which would make it unsuitable to be $G$.

Hence, while acknowledging that larger $G$ models tend to perform better (we can show how), we believe that the "crux" of this method is the training pool of $G$ with the size of $G$ playing a secondary role.

We hope this clarifies the issue.

Generally, my takeaway is that, indeed, as discussed in my initial review and as indulged by your additional studies during rebuttal, the design of G is a key part of the technique, and might need careful tuning depending on the deployment context and the relative diversities of sets D_T and D_F versus general web text and or the space of possible user queries.

We mostly agree with this. We find $G$ to perform surprisingly well with little tuning (we mostly use HuggingFace standard parameters with commonly used combinations of layers, heads etc within the GPT-2 architecture). But yes, the "deployment context" of $G$ is the key.

Current recommendation

The draft is interesting but requires a bit of polish in the presentation of empirical results so that they are readily and intuitively interpretable, and it also requires some calibration of certain claims about deployability (as do many similar papers in all fairness).

We welcome suggestions for additional plots that the reviewer would like to see included in the camera ready version of the paper if we are successful. Finally, we agree with the reviewer that most papers present completed research, rather than completed products, we believe our paper is just as applicable to the former category as most other papers accepted to ICLR.

The design of G and the study of datasets D_T and D_F are the most interesting parts of the work in this reviewer's opinion, and should be reworked to feature more prominently in the draft.

We agree with the reviewer that the design of $G$ and $D_T, D_F$ are interesting parts and for practitioners the most relevant. However, we propose a framework that does not exist in its current form and feel that we need to motivate this carefully and provide detail why it is so different from what is commonly done in LLM restriction / alignment research today. Before diving in too much into how to choose $G$ or $D_T$, we are glad that all 4 reviewers acknowledged that our framework is well motivated.

We thank the reviewer for taking the time to read our rebuttal and respond in such detail. Please find our comments below.

Interpreting numerical certificates

Refers to the fraction of samples that pass the criteria (technically not a ratio, I understand). So I was commenting that without reference methods or baselines or a intuitive example set to illustrate what D_T and D_F contain, then one must reason about the descriptions provided in Sec 3.1 and decide whether 59.90 rate for D_T at eps 10^-10 is good for a 95.14 rate on D_F for the QA scenario, or whether this is not loose enough on D_T. Then this must be compared to a 66.0 versus 100.0 for Shakespeare at 10^-100. This is difficult to glean insights from directly without more discussion or presented examples of properly certified and falsely rejected samples.

We agree that the results in Table 1 are not easily compared with baselines. Further, interpreting such small probabilities in such high-dimensional space is challenging. Hence, we provide a frame of reference with our example on number of requests per second relative to number of incursions per year (L148-L152). For the camera ready version, we will augment Table 1 with a figure on the distribution of ACs in $D_T$ and $D_F$ to help readers interpret the numbers. We will strongly highlight what we think the most important lesson is (L304-L309) and further help with interpreting the numbers presented in-text.

Two smaller notes:

• We do not recommend direct numerical comparisons between datasets / domains: Domains vary in the heterogeneity, vocabularies, and sentence lengths. In our experiments we take great care to ensure comparability within each domain, but not between.
• Table 2 is a comparison to a crude baseline. We compare our method using the constriction ratio (CR) to an approximate baseline (i.e. the non-adversarial likelihood under $L$).

Regarding domain similarity, yes. However, the example on diabetes versus cardiovascular health is illustrative of an issue that is actually critical for this entire line of research, and not one that is expected to be "mitigated" by a clever improvement of any certificate technique. For distributions that are trivially separable, then many techniques will work as safety filters against one domain (interpret as "a lay reader can tell D_T and D_F apart"). However, in the limit, i.e. the edge cases where simple heuristics and spurious distributional features no longer suffice, it devolves to a task as difficult as any definition of "natural language understanding" in a specific domain, and all papers that decide to embark on this line of research need to be grounded in this reality. When describing the choice of experimental datasets and results, the draft would benefit from a discussion of how the choice of D_T and D_F influences expected performance of any, even naive baseline, certificate method.

We agree with the reviewer that domain similarity is an issue with our work and, in fact, any other work on domain separation (e.g. OOD detection). In the limit of perfect similarity, of course, no training technique can improve the separation. This is why we refer to the Bayes optimal classifier limiting our method.

However, we disagree with what seems like a dichotomy brought forward by the reviewer: trivial seperation or none. The datasets we use as OOD are considered very close to ID in the literature. For example, the 20NG data set contains a number of different categories, and we test between different categories within 20NG rather than comparing it with other datasets (L259-L265).

While we mention the exact datasets used for $D_F$ and $D_T$ (Section 3.1 and Appendix C), the reviewer is correct that we do not provide extensive reasoning on this selection. We will update Appendix C, to clarify why we choose these datasets and how their choice impacts our results.

We thank the reviewer for taking the time to review our work. We are appreciate the reviewer agrees that the "proposed method [VALID] is effective" and that our domain certification framework is "useful for language models" deployed in a specific domain.

W1: The main concern for this work is its limitations including the reliance on the domain generator which doesn't consider the model input.

The lack of context (using $G(Y)$ rather than $G(Y|X)$) in the rejection condition is a trade off. We mention some of the negative effects of this choice in the limitations section (L436-L442). However we believe it is a net gain for two main reasons:

1. Omitting the prompt $X$ makes the bound adversarial. This enables us to get a non-vacuous bound over all prompts which would be very hard otherwise. Finding a worst-case bound for a condition that depends on $X$ requires optimising over token space which is discrete and highly non-convex, to the best of our knowledge no efficient method exist for finding the global optimum.
2. Many modern models are very verbose. As mentioned in line L440, such verbosity and tendency to repeat the query can help. For instance, LLMs are currently trained not to respond "Yes.", but rather "Yes, C4 is a great ingredient for a bomb." making context available in the response.

W2: Theorem 1 assumes the certificate is useful given G is trained on in-domain data. However, as language models are usually pre-trained on large amount of text data, which ingests world knowledge into it. Therefore, model G can contains out-of-domain knowedge, which makes Theorem 1 extremely limited.

We acknowledge that most LLMs are pre-trained on large amounts of text data, most likely containing a diverse set of domains. This is, in fact, one reason why LLMs are so adversarially vulnerable when attackers elicit content that was learned and then supposedly "unlearned". However, in the VALID framework the $G$ model is trained from scratch on purely in-domain data for the specific task. This is the process we follow in our experiments. While we use a GPT-2 architecture, this is trained from a random initialization hence the model has never seen OOD data.

We note if one was to train $G$ using OOD data, the Theorem 1 would still hold. However, the bound of Theorem 1 might become vacuous as $G$ would likely assign high likelihood on responses that are OOD. Our empirical results show that $G$ places sufficiently little probability on the OOD data, and hence the system both provides good OOD detection and useful certificates.