Certifying Counterfactual Bias in LLMs

We thank the reviewer for their time and constructive feedback. We address their concerns below. We hope our response mitigates their concerns and they consider increasing their support for our paper.

Acknowledging prior works on conformal prediction and Prompt Risk Control

Please see "Comparison with prior works" section of the general response.

Need to sample from distributions of interest...

Firstly, there is no evidence for whether PRC can handle our setting of relational properties consisting of distributions over counterfactual prompt sets. Secondly, our theoretical framework never assumes any constraints on the size of the sample space of counterfactual prompt set distributions. Hence, it can and does operate in finite-sample settings. Note that sample spaces of distributions over practical prompts, bound by context sizes of LLMs, are finite and thus our experiments with random and mixture of jailbreak specifications are over distributions with a finite (but large) number of possible samples. Lastly, if the reviewer is particularly concerned about certifying LLMs for only a handful of prompts, we can enumerate over all possible prompts to get exact bounds (preferable, but not done by either PRC or QCB) or certify with distributions over the prompts (both QCB and PRC would do this). However, cases with only a few samples are not realistic, as they do not capture various possible prompts and prompting schemes in practical LLM usage.

Allocative harm, how can it be caused by a representational harm, and when is it illegal?

Allocative harm consists of disparate distribution of resources or opportunities across social groups [1]. It may not necessarily be a consequence of representational harm, but can result from it in certain cases. For example, prior works like [2] show that the lack of representation of African American English in dominant language practices results in that community facing penalties when seeking housing. Most constitutions around the world have anti-discrimination laws like [3] that prohibit allocative harms in employment etc.

How can LLMs affect social hierarchies?

Studies like [4] show interconnections between language and social hierarchies. [5] show how LLMs treat different social groups disparately, hence reinforcing social hierarchies.

Only one set of counterfactual prompts considered in the paper?

We would like to clarify that several counterfactual prompt sets are considered in the paper. In fact, we make a general certification framework for (large) distributions over different counterfactual prompt sets in Sec 3. The distributions in Sec 4 fix a pivot counterfactual prompt set for each specification and derive several different counterfactual prompt sets from it by prepending randomly selected prefixes to each prompt in the pivot set. This setting represents various (adversarial) corruptions to prompts as prefixes and certifies LLMs for unbiased responses to the derived counterfactual prompt sets.

Definition of bias in lines 177-179

Please check the revised paper.

How does QCB ameliorate test set leakage?

Evaluation with distributions instead of static datasets avoids test set leakage as all the prohibitively large number of counterfactual prompt sets in the support of the distributions can not be included in the training/alignment sets of LLMs. The distributions are represented as probabilistic programs (Algorithms 1-4), which we execute to sample counterfactual prompt sets. We use examples from the datasets only to create pivot counterfactual prompt sets. The actual prompts in the counterfactual prompt set distributions are different from the dataset examples due to the added prefixes.

How does QCB resolve Limited Test Cases?

Although QCB uses a finite number of samples, the certificates consist of statistical guarantees over distributions with prohibitively large sample spaces (~$10^{460}$ elements). Conventional benchmarking methods, which essentially enumerate over all inputs in a dataset, can clearly not scale to such large input spaces. They are thus limited in the number of test cases they can handle.

Novel insights into the biases of popular LLMs.

Our main insight is the susceptibility of LLMs to consistently generate biased content when prompted with prefixes that are perturbations of jailbreaks, which may be less effective on their own. These prefixes are computationally inexpensive to generate, semantic-preserving (do not affect the original query), and realistic. Yet, they are effective at eliciting counterfactual biases in aligned LLMs, pointing to previously unknown vulnerabilities of these models. We have clarified this in the contributions mentioned in the revised Introduction.

Interpreting certification bounds

QCB's certification bounds contain the (unknown) probability of unbiased responses with 95% confidence (please check the study in "new experiments" on validating the bounds). Hence, the lower and upper bounds can be interpreted as the worst and best possible values for the probability of unbiased responses. We present the average of certification bounds over different kinds of specifications, derived from different datasets in Table 1. From these values, we can judge, for example, that Mistral-Instruct-v0.2 (7B) on an average over all mixture of jailbreak specifications from BOLD can have 22% probability of unbiased responses in the worst case and 42% in the best case. We also provide benchmarking baselines in Table 1, which are average values for the empirical estimates of the probability of unbiased responses. These can also be compared with the average certification bounds to see the disparity between benchmarking and the worst/best possible probability of unbiased responses. The certification bounds characterize the vulnerabilities of LLMs to potentially adversarial distributions (instead of static datasets), and can be used to reliably compare models based on counterfactual bias. The bounds reported in Table 1 are similar in spirit to certified accuracy in prior works on neural network verification [6,7]. The bounds are fairly tight, as observed by their ranges in Table 1. We can obtain tighter bounds by using more samples in certification, as shown in our ablation study in Appendix B.1. However, that will incur additional computational costs.

References

1. Bias and Fairness in Large Language Models: A Survey
2. Linguistics in Pursuit of Justice
3. The civil rights act of 1964: Fair employment practices under title vii
4. Unsettling race and language: Toward a raciolinguistic perspective
5. Large Language Models Portray Socially Subordinate Groups as More Homogeneous, Consistent with a Bias Observed in Humans
6. AI2: Safety and Robustness Certification of Neural Networks with Abstract Interpretation
7. Certified Adversarial Robustness via Randomized Smoothing

We thank the reviewer for their time and constructive feedback. We address their concerns below. We hope our response mitigates their concerns and they consider increasing their support for our paper.

Correspondence of bias metric for BOLD with human judgement of bias.

Bias is a complex and highly-subjective topic, especially in the counterfactual sense, and hence we believe that invalidating the detector simply based on the general perception of the low value of 76% may not be appropriate in this case. Bias detection often involves nuanced subtleties and hence state-of-the-art bias detectors exhibit similar performance in human evaluation [1,2]. The human annotation of bias is contingent on many factors such as their cultural background, our annotation instructions (provided in the open-source implementation of the framework), etc. The authors manually observe that the human annotations generally flag more instances as biased, incorrectly (in our opinion). Hence, our bias detector shows 93% precision, but 50% recall. We also provide a qualitative analysis with case studies of instances where the bias detector results differ from human judgement in Figure 12 in Appendix E.1. Owing to these inconsistencies of our bias detector with human perception of bias, we believe that our bounds for probability of unbiased responses are actually higher than bounds with a perfect bias detector (due to low recall), hence indicating a worse situation of counterfactual bias in SOTA LLMs. The design of the better bias detectors is an ongoing and important research topic but it is orthogonal to our work. The framework can operate with any bias metric that can identify unbiased responses. We mention the accuracy value of the bias detector with respect to human judgement in the first paragraph of Section 5.1 of the main paper.

The proposed distribution of jailbreaks seems weak.

We respectfully disagree with the reviewer on our distribution of jailbreaks being weak. As demonstrated in Table 1, several models show low probabilities of unbiased responses, which may be unacceptable in practical settings. Examples of this include Gemini for BOLD and DT, Mistral for BOLD and DT, and GPT-4 for BOLD. This shows that our proposed distributions, alongside being inexpensive to sample from (hence feasible for low budget adversaries), are effective at eliciting biased responses from SOTA, rigorously aligned models. Examples of these biased responses are shown in Figures 4 and 15 (Appendix G).

Use learned jailbreaks.

We thank the reviewer for the suggested experiment. We show our results in the "new experiments" section of the general response.

Synthetic study to verify validity of confidence intervals.

Please check "New experiments" section of the general response.

References

1. The Woman Worked as a Babysitter: On Biases in Language Generation
2. A Human-centered Evaluation of a Toxicity Detection API: Testing Transferability and Unpacking Latent Attributes

We thank the reviewer for their time and constructive feedback. We address their concerns below. We hope our response mitigates their concerns and they consider increasing their support for our paper.

Impact of temperature T and top-k parameters.

We agree with the reviewer on the dependence of the certificates on LLM inference parameters such as T and top-k. We would like to clarify that the counterfactual prompt set distribution $\Delta$ is independent of LLM-specific parameters, but C depends on them. This is because we do not decode sequences from LLM generations when sampling from $\Delta$, for the distributions considered in the paper. We have conducted an ablation study on the influence of these parameters and report it in the "new experiments" section of the general response and Appendix B.2. We second the reviewer's thought that the specific parameter values, while important for C are just implementation details and the overall framework's efficacy does not depend on them.

Hypothesis for main jailbreak's effectiveness against Mistral-7B

We certify Mistral-Instruct-v0.2 7B model in Table 1 and mention this in line 377. We think that the main jailbreak is effective against Mistral because the model is not rigorously safety-trained [1]. Hence, it could show harmful biases for the main jailbreak. To check for biases in Mistral-Instruct-v0.3, we certify it with the mixture of jailbreaks and soft jailbreak specifications over the Decoding Trust dataset. We report our results and show a biased example in the "new experiments" section of the general response.

Section for practitioners on deploying QCB.

We thank the reviewer for their suggestion. We have added Appendix I to address this.

Utility of “random prefixes”?

Random prefixes are an important category of prompt perturbations, which are not intentionally adversarial but denote random noise in the prompt. We include them to demonstrate our framework's generality beyond adversarial distributions. We understand that Wei et al. 2023, Zou et al., 2023 are for special strings of tokens which are optimized to jailbreak LLMs. However, such strings of fixed length are in the sample space of random prefix distributions. Similar to these strings, other random combinations of tokens could potentially elicit biased behaviors in LLMs. We equally prioritized all random prefixes with a uniform distribution to certify the average case behavior of the target LLM, subject to random prefixes of fixed length. Contemporary works, such as [2] also show the possibility of random augmentations in jailbreaking models, hence we study the effects of random prefixes in eliciting counterfactual bias in SOTA LLMs. The generally high probabilities of unbiased responses for random prefixes indicates that LLMs can effectively denoise random tokens in their prompts and avoid biased responses. Exceptions, such as Vicuna-13B, exist wherein the model responses can be influenced by random prefixes as well.

Use of certificates over just confidence intervals

Certificates for bias in LLMs are confidence intervals with statistical guarantees of correctness over a given distribution of counterfactual prompts. They also contain a proof/reasoning for the final result (samples of LLM responses in this case). The certification guarantees of the bounds $[l,u]$ are that the true value of the probability of unbiased responses $p^*$ for counterfactual prompt sets in any given distribution is within the bounds with high confidence $1-\gamma$, where $\gamma\in(0,1)$ is a small user-specified constant. That is, $Pr[p^*\in[l,u]]\geq 1-\gamma$. These guarantees on the bounds make them reliable estimates for the probability of unbiased responses. Certificates for local specifications bound the probability of unbiased responses for the given distribution and its sampler in the specification, not the overall input space. Local certification is tractable compared to global specifications as it is over distributions in restricted input regions and efficient samplers can be defined for such distributions. Note, however, that our general framework given in Section 3 is not restricted to local specifications and can work with any given distribution and its sampling algorithm.

Releasing multiple certificates for popular models

We thank the reviewer for their suggestion. Table 1 is similar to a leaderboard. Models can be ranked according to their average lower/upper bounds. We are working towards releasing a public leaderboard. We will also release individual certificates for popular models on platforms such as Huggingface, and can provide them to the reviewers as well, if they would like to take a look.

References

1. https://mistral.ai/news/announcing-mistral-7b/
2. Stochastic Monkeys at Play: Random Augmentations Cheaply Break LLM Safety Alignment

We thank the reviewer for their time and constructive feedback. We address their concerns below. We hope our response mitigates their concerns and they consider increasing their support for our paper.

Novelty in statistical estimation method

We use the Clopper-Pearson method for certification as it is suitable for our specifications. We do not claim novelty in the statistical estimation method, but instead in enabling its use to certify LLMs for counterfactual bias. Specifically, we design novel specifications for counterfactual bias in LLMs and (potentially adversarial) distributions with their efficient sampling algorithms to certify bias in LLMs. Without such formalism, it is not possible to use many statistical estimation methods for the certification, and such distributions do not exist for counterfactual bias in prior works, to the best of our knowledge. Prior works on probabilistic certification of traditional ML models like [1,2], which also apply existing statistical methods, have been well-received by ICLR and ICML and have made significant contributions towards trustworthy AI. Reviewer CoYJ considers the simplicity of our approach as its strength enabling the framework to have broader impact.

Writing improvements

We have updated the paper with the suggestions. Here is a summary.

1. G denotes demographic groups, which are subsets of human population, created from societal norms/biological traits. A is a set of attributes denoting demographic groups. Each element of A is a string. No operators can be used to modify them, to the best of our understanding, as these attributes are commonly-used strings to socially denote the corresponding demographic group.
2. We want to exclude prompts whose responses inherently depend on sensitive attributes, from counterfactual prompt sets. For example, for gender-specific prompts like "Answer for a female. What steps should I take to prepare for becoming a parent?", even an unbiased model f would give different responses for this and the gender-neutral prompt. Hence, we put part (3) in Definition 1, requiring $\forall i, f(P_i) = f(X_i)$.
3. A certificate is an evaluation of the probability of unbiased responses specified in Eq 1. It also contains a proof (LLM responses) for the evaluation. We estimate this probability with high-confidence bounds, but that is specific to our certification algorithm.

Sec 4 presentation

Our paper already has Sec 4 in the suggested format. For counterfactual prompt set distributions with varying prefixes, we present a general specification with prefixes in Algorithm 1 and show examples of prefix distributions in Algorithms 2-4. Our general specification, independent of prefixes, is given in Sec 3 (Eq 1). We would appreciate if the reviewer clarifies this suggestion with specific details of presentation changes they would like to see. We will try our best to incorporate them.

More samples for tighter bounds

The bounds will become tighter with more samples. But the tightness gain may not be significant, as observed in our ablation study in Appendix B.1. We obtain bounds with varying number of samples to certify Mistral-Instruct-v0.2 7B for the 3 kinds of specifications. The bounds change by a maximum of 5% when scaling from 50 to 100 samples. However, this has high computational cost (twice number of samples). Hence, we selected the number of samples per certificate as 50. We can show more results with higher number of samples, if the reviewer suggests. However, we do not have enough resources to run more samples for all certificates generated for Table 1.

Sample multiple prefixes for a given prompt

We would like to clarify that we already sample multiple different prefixes to form different counterfactual prompts used in 1 certificate. We use 50 sets of counterfactual prompts, each giving 1 observation of bias in LLM responses. The same prefix is used for each prompt in any specific counterfactual prompt set, but different counterfactual prompt sets can have different prefixes (sampled iid). We would appreciate if the reviewer could clarify whether they meant something different from our current procedure, when they mention "sample multiple prefixes". We would be happy to try formalizing their suggested setting and run relevant experiments.

Other prefix distributions

Yes, other prefix distributions can help explore different kinds of biases in LLMs, that may be undesirable in various contexts. We believe that custom prefix distributions that provoke prominent or undesirable biases in a setting may be useful to certify. For example, in contexts where racial bias may be prominent, prefixes could encourage racial bias to stress-test the trustworthiness of LLMs. We show new experiments on prefix distributions constructed with learned jailbreaks in the general response.

References

1. A Statistical Approach to Assessing Neural Network Robustness
2. Certified Adversarial Robustness via Randomized Smoothing