An Auditing Test to Detect Behavioral Shift in Language Models

• [..I would recommend giving a clear formulation for such two cases before stepping into examples..]

  • Thanks for this. We now give a clear description of both the “internal auditing” and the “external auditing” use cases in Section 1.

• [..The authors may elaborate more on the design of [the tolerance parameter] to enhance the generability of the proposed method.]

  • The tolerance parameter epsilon is a hyperparameter of our test that has to be set by practitioners based on their use case.
  • With experiments involving a wide range of behaviors, datasets and models, we think it is possible to provide a more general guideline on what constitutes a “small”, “medium” or “large” neural net distance. This being out of scope here, we instead suggest some practical strategies to set the hyperparameter epsilon. We clarified this reasoning at the start of Section 4.2.
  • In particular, we wanted to explore 1) a “conservative” strategy to set this hyperparameter that might be appropriate in a safety-critical setting where we want to detect even very small differences in distributions. And 2) a “liberal” strategy to set the hyperparameter where only changes larger than some significant distance are detected.

• [..‘To determine a tolerance parameter $\epsilon$ for this toxicity setting, we consider triggering the test if a model differs by more than the aligned Llama3 model with different sampling parameters.’ Is there any rationale to support such a design?..]

  • For this “conservative” setting, we chose epsilon as the distance between the distributions induced by the base model with two different sampling strategies. The rationale behind this is that we might consider these changes to be relatively insignificant.
  • The concrete choice of the alternative sampling parameters was informed by practical knowledge and made with two considerations in mind: (1) The alternative sampling parameters should be "realistic" and not be extreme enough to cause the model to only output gibberish and (2) They should be different enough from the default to cause some change in the model’s behavior.
  • We added this context for the sampling parameters in Appendix A.2.

• [..The experiment design for the false positive rate in Section 4.1 is somewhat rough and lacks details..]

  • In our experiments in Section 4.1 on the false positive rate, we created two sets of generations from the same model with the same sampling parameters and using the same prompts - but with different random seeds. The two individual continuations of the same prompt (and thus individual behavior scores) can be slightly different. However, both of the model generations (and thus behavior scores) come from the same distribution.
  • Since those underlying distributions are identical, we can use this setting to investigate the false positive rate of our proposed method. We revised Section 4.1 to make the rationale behind this set of experiments clearer.

• [..A short description of the five instruction-tuned models can benefit the readability of the paper as they are used in most experiments..]

  • To investigate whether fine-tuning on "unrelated" tasks affects toxicity, we produce 5 versions of Llama3-8B-Instruct by instruction-tuning on subsets of the SUPER-NATURALINSTRUCTIONS dataset ([1]).
  • We now describe these instruction-tuned models and how they are trained in more detail in Section 4.2 as well as in Appendix A.

• [..For the 2nd use case (internal audit) ... Why do you use the neural distance between models with simple prompts and few-shot prompts?..]

  • For the “liberal” strategy, we chose epsilon as the distance between the baseline model and the same model with few-shot prompting. The rationale behind this is that a distribution difference that is smaller than what can be achieved just with more sophisticated prompting and no underlying model change might not be considered significant.
  • We emphasize that of our strategies for the "internal" and "external" auditing setting are suggestions for how to set epsilon - a practitioner might devise different strategies for their use-case; note also that epsilon can be set to 0 if there is no justifiable lower bound for a change that should be tolerated.

• [..The authors provided the link to the replication package, but the link seems incomplete..]

  • The current link can be seen as a placeholder to our repository. We will update it in case of acceptance.

• [..there are still some typos and grammar issues..]

  • We reviewed the paper for typos and grammatical errors.

[1] Wang, Y., Mishra, S., Alipoormolabashi, P., Kordi, Y., Mirzaei, A., Arunkumar, A., ... & Khashabi, D. (2022). Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks. arXiv preprint arXiv:2204.07705.

Dear Reviewer grRY,

Thank you for your thoughtful feedback on our submission. We have tried to carefully address each point raised in our response and we have made corresponding improvements to the paper. We would be grateful if you could review our revisions and responses to your comments. If you have any remaining questions or concerns, we welcome your additional feedback.

-The Authors

• [..How the method scales with larger or more complex models is unclear..]
  • We agree that evaluating our method using larger and more complex LLMs would make our results even more robust. We address this by including Llama-3-70B-Instruct as a fourth baseline example. We focus on Use Case 2 (Internal Audit, Translation Performance) from Section 4.2, as time constraints preclude us from producing instruction-tuned versions of Llama3-70B-Instruct. We compare the BLEU score distributions of Llama-3-70B-Instruct against Aya-23-35B, using few-shot prompted Llama3-70B-Instruct to determine a tolerance threshold $\epsilon$. We find that few-shot prompting has a strong effect on the translation performance of Llama3-70B-Instruct, while Aya-23-35B does not meaningfully change the distribution further. We added this finding in Appendix 4.2.2.
• [..The presentation is not good. Some parts are hard to follow..]
  • We included a definition of BSA in the abstract and elaborated more on our use cases in Section 1.
  • We streamlined Section 2, removing unnecessary details.
  • In Section 3, we introduced concepts in a way that builds upon each other for better understanding.
  • We enhanced the presentation of the experiments, especially in Section 4.2.
• [..There are some typos scattered around..]
  • We reviewed the paper for typos and grammatical errors.
• [..Section 1: ‘without tipping off a potential bad actor’ Why is this a requirement of behavioral shift auditing?..]
  • We agree that this condition is not central to our use-cases and have removed it to improve clarity.
• [..Section 1: What are the main contributions of your work?..]
  • We added a paragraph summarizing our contributions to the end of Section 1: “Our key contributions are: (1) formalizing Behavioral Shift Auditing and developing a statistical test for detecting LM behavior changes from model generations, (2) providing theoretical guarantees on false positive control and test consistency, (3) introducing a configurable tolerance parameter enabling both strict external audits and flexible internal monitoring, and (4) demonstrating effectiveness through toxicity and translation case studies showing detection with hundreds of examples.”
• [..Section 2.2: The high-level introduction of anytime-valid hypothesis testing is not easy to understand..]
  • Thanks for this. We removed some unnecessary detail from Section 2.1 and 2.2 that might be confusing to the reader and instead spent more time explaining the fundamental concepts of (deep) anytime-valid testing to make them more intuitive.

Dear Reviewer 5LeM,

Thank you for your thoughtful feedback on our submission. We have tried to carefully address each point raised in our response and we have made corresponding improvements to the paper. We would be grateful if you could review our revisions and responses to your comments. If you have any remaining questions or concerns, we welcome your additional feedback.

-The Authors5LeM

• [..Behavioral-shift auditing is vague and is not properly defined..]

  • Thank you for this. We have added a definition of BSA in the abstract: “We present an efficient statistical test to tackle Behavioral Shift Auditing (BSA) in LMs, which we define as detecting distribution shifts in qualitative properties of the output distributions of LMs.” and also in section 1: “We call the general class of problems detecting changes in LM behavior distributions Behavioral Shift Auditing (BSA) problems.”

• [..The abstract and first two paragraphs of the intro do not make it obvious that the behavioral test is a statistical test..]

  • We now explicitly state that our proposed test for BSA is a statistical test in the abstract and section 1.

• [..It’s not clear why practitioners would want ‘to detect changes without tipping off a bad actor’..]

  • We agree that this condition is not central to our use-cases and have removed it to improve clarity.

• [..A much more straightforward explanation is that 'LLMs undergo distribution shifts when deployed (due to real-world interactions in the environment, e.g., [1]), driving a need to continuously monitor for shifts in their behavior..]

  • Our sequential testing approach could be used to monitor for distribution shifts in LLMs when deployed and we would be excited to explore this more in the future. In the current work, we focused on detecting when the behavior of a model changes due to some internal changes, e.g. because of fine-tuning or a model being swapped. Our proposed test is specifically constructed for this problem, comparing the two model's generations on the same data.

• [..the qualitative example of the imaginary company in L46 - 72 comes out of nowhere and is confusing..]

  • We removed the example and instead elaborated more on the “internal auditing” and the “external auditing” use cases. We hope this reduces confusion and strengthens our motivation.

• [..The paper makes a strong assumption about the availability of an empirical classifier for behaviors..]

  • (See also response to reviewer Y3yQ): We agree that the design of accurate behavior scoring functions can be difficult. We think that the use of other LLMs for this might be promising (e.g., [1], [2]).
  • We also note that our proposed test can tolerate some noise in the behavior scoring function. We conduct an experiment to investigate the effect of a noisy behavior scoring function on the detection rate. Results presented in Appendix C.4 show that it takes our test longer to detect a distributional difference the noisier the behavior scoring function is, but also that detection rates still converge to the same level, demonstrating our method's robustness.
  • Even more so, we claim that even a scoring function that is just correlated with the true behavior score on the test data provides some information about a potential shift in distribution. For example, the BLEU score that we have used as a metric in section 4.2 can be seen as such a proxy for translation performance. See Appendix C.4 for further discussion.

• [..it is difficult to evaluate certain behaviors, such as deception [4] or sandbagging [5], from a single (prompt, completion) pair, as they tend to occur over an entire trajectory. Furthermore, evaluating the mere presence of other behaviors, such as hallucinations [6], is an active area of research..]

  • Thanks for these examples. We have added a paragraph discussing these and the above difficulties involved in using a behavior scoring function in section 6.

• [..Second sentence of the abstract has an extra T: T This --> This..]

  • Typo was removed.

• [..In figure 5, why is the orange dashed line not at x-value (test epsilon) which corresponds to a y-value (proportion of triggered tests) of 0?..]

  • We have improved the clarity of the writing in section 4. This includes a more informative caption underneath figure 5.
  • Dashed lines depict our estimate of the true neural net distance between the model in question and the baseline model, while curves depict the proportion of positive test cases. We find that for the model depicted in orange, a Llama3-8B-Instruct model instruction-tuned on the task categories of Gender Classification, Commonsense Classification and Translation, the false positive rate is larger than our alpha-level of 5% in two cases. This can happen for two reasons: First, our estimate of the true neural net distance contains some noise. Second, we repeat our test "only" 24 times. Given an $\alpha$-level of $\rho<=0.05$, the probability that more than 5% of those 24 runs falsely show a positive result can be as high as 0.34.

[1] Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., & Zhu, C. (2023). G-eval: Nlg evaluation using gpt-4 with better human alignment. arXiv preprint arXiv:2303.16634. [2] Fu, J., Ng, S. K., Jiang, Z., & Liu, P. (2023). Gptscore: Evaluate as you desire. arXiv preprint arXiv:2302.04166.

• [..it is reasonable to expect large-scale implementations of $B$ to involve an ML model and therefore add noise…A deeper discussion and empirical investigation into the sensitivity of the proposed method wrt different types of $B$'s would strengthen the paper..]

  • Thank you for this. We agree that the design of accurate behavior scoring functions can be difficult and will likely involve ML models. To highlight this we have added a paragraph discussing the difficulties involved in using a behavior scoring function in Section 6.
  • Our proposed test for Behavioral Shift Auditing (BSA) is robust to some noise in the behavior scoring function. Specifically, as long as the expected output of the noisy scoring function corresponds to the true behavior score – that is, $B(\mathbf{x,y})=\mathbb{E}[\tilde{B}(\mathbf{x,y})]$ for every $(\mathbf{x,y})$, where $\tilde{B}$ is the stochastic behavior scoring function and $(\mathbf{x,y})$ denotes a prompt-generation pair – the statistical guarantees of our test remain valid in the limit.
  • However, we agree that an empirical investigation into the effects of a noisy scoring function is important as the noise might make it harder for the betting score network to learn. To investigate this, we conducted an experiment (summarized in Appendix C.4) where we added varying levels of Gaussian noise to behavior scores. The results show that it takes our test longer to detect a distributional difference the noisier the behavior scoring function is, but also that detection rates still converge to the same level. This sensitivity study gives us insight into the robustness of the proposed test for BSA, thank you for suggesting this.
  • In Appendix C.4, we have also added a deeper discussion about the effects of certain types of systematic errors in the behavior scoring function on test results.

• [..a practitioner might consider it insufficient for the model to pass a single test of behavior…It's unclear how the proposed method would handle multiple behavior objectives (e.g., low toxicity and low hallucinations and ...)..]

  • We agree that a practitioner might want to test for changes in multiple behaviors, and a useful prompt dataset would exhibit each behavior to be tested (so that both the propensity to hallucinate and toxicity could be evaluated in the model generations). Our test can be naturally extended to this case:
  • For the exact test, the extension is straightforward, as it is an application of Deep Anytime-Valid Testing (DAVT), which has been successfully applied to multi-dimensional distributions [1].
  • For the tolerance test, one can either set a global tolerance hyperparameter as the maximally-allowed distance between multidimensional distributions, or conduct multiple tests in parallel with individual thresholds for each behavior. We discuss this extension in Appendix C.5 and briefly mention it in Section 6 of the main text.

• [..Is it possible to construct a baseline by simply comparing $B$'s values on the two models' outputs using test data? ..]

  • Thank you for this. We had a similar difficulty finding a baseline for this exact problem. Our thinking was that since our work is an extension of Deep Anytime-Valid Testing (DAVT), and they have conducted a detailed comparison with other testing baselines and found their test to generally outperform those baselines, that this was the most informative comparison [1].
  • Because of this we focussed our effort on sensitivity studies for BSA.
  • We also report mean toxicity scores for different LMs in Figure 8 in Appendix A.2 and mean BLEU scores in the main text, Section 4.2 line 416/417 (“Few-shot prompting leads to a modest increase in mean BLEU scores from 0.1683 to 0.1765.”). Wasserstein distances between instruction-tuned checkpoints and baselines can be found in Figure 2 and 4, as well as Figure 10 in Appendix C.1.
  • However, while summary statistics (e.g., mean and variance) and distance measures (e.g., Wasserstein distance) are commonly used to compare distributions, they may not capture critical aspects of behavior distributions or provide clear decision rules for detecting changes.
  • For completeness we have also added new results evaluating the classical two-sample Kolmogorov-Smirnov (KS) test. Unlike our anytime-valid method, classical tests such as this can suffer from inflated Type I error rates when used repeatedly as new data is added. This can lead to increased false positives and unreliable conclusions. In Appendix C.3, we compare our test to the KS test and show experimentally that our method avoids the pitfalls of increased Type I errors, highlighting its robustness and practical advantages over traditional methods.

[1] Pandeva, T., Forré, P., Ramdas, A., & Shekhar, S. (2024, April). Deep anytime-valid hypothesis testing. In International Conference on Artificial Intelligence and Statistics (pp. 622-630). PMLR.

Dear Reviewer Y3yQ,

Thank you for your thoughtful feedback on our submission. We have tried to carefully address each point raised in our response and we have made corresponding improvements to the paper. We would be grateful if you could review our revisions and responses to your comments. If you have any remaining questions or concerns, we welcome your additional feedback.

-The Authors