Prompt Risk Control: A Rigorous Framework for Responsible Deployment of Large Language Models

Thank you for your feedback on our submission and recognizing its potential for significant impact on responsible LLM deployment. Below we provide our responses to your remaining concerns. We hope that you would consider raising your score if our responses adequately address these concerns.

Q: As the authors mentioned, the risk constraints may not always be satisfiable using this framework.

A: We believe that if an acceptable risk bound cannot be guaranteed, it is best that this is identified and made clear to all stakeholders, whether or not the LLM is still deployed.

Q: How about the training cost of the proposed framework? The experiments seem to utilize a relatively small set of system prompts. Would using a larger set yield different results?

A: There is a trade-off between the number of candidate prompts and the tightness of the bounds. We will make this more clear in our writing.

Q: From my perspective, solely selecting the system prompt may not be the ultimate solution for ensuring the safe output of LLMs. Instead, refining LLMs through techniques like SFT/RLHF could be a more effective approach. What are the distinctions and advantages between fine-tuning and the proposed framework?

A: We believe that PRC is complementary to these other approaches. We use models prepared with instruction tuning in our experiments, and mention how our framework can be applied to models like GPT-4 which likely undergoes RLHF. We only require that the loss values are produced after the LLM is fixed, and are agnostic to any development and refinement process applied to the LLM prior to that. PRC should be just one tool in a set of techniques for enabling responsible LLM deployment. Also, we note that our algorithm is much less resource intensive than SFT or RLHF.

Q: Can this proposed framework effectively counter adversarial attacks?

A: We propose an application via red-teaming in Section 5.2.1 as an example of how PRC can be used to understand vulnerability to adversarial attacks.

Thank you for your thoughtful and important feedback on our submission. Below we respond to the particular issues and questions that you raised. We hope that you may consider raising your score if our responses sufficiently address your remaining concerns with the research.

• We would like to highlight the novelty and importance of our algorithm for bounding QBRM and statistical dispersion measures under distribution shift. Work like [1-3] has been published at top venues for significance w.r.t. offering bounds on a much more limited set of measures in the covariate shift setting. We believe that our submission offers a significant technical contribution by enabling the bounding of an extremely rich and important family of risk measures.

  • [1] Sangdon Park, Edgar Dobriban, Insup Lee, and Osbert Bastani. Pac prediction sets under covariate shift, 2022.
  • [2] Ryan J. Tibshirani, Rina Foygel Barber, Emmanuel J. Candes, Aaditya Ramdas. Conformal Prediction Under Covariate Shift, 2019.
  • [3] Hongxiang Qiu, Edgar Dobriban, and Eric Tchetgen Tchetgen. Distribution-free prediction sets adaptive to unknown covariate shift, 2022.

• While we have made our best effort to design good prompts for each experimental task, how to best engineer prompts is not a focus of this work. Rather, we aim to mitigate the risk and democratize the process of writing and selecting prompts, so that it is based on rigorous risk bounds instead of assumed expertise or optimizing simple empirical averages on a validation set.

• Our experiments are meant to illustrate how one might choose contextually relevant risk measures based on the application. For example, in societally important domains like medicine one might be concerned with uneven dispersion of loss across patients, which is captured by the Gini coefficient, while in code generation the only concern may be the rate at which the code is valid. On the other hand, for chatbots there may be multiple loss and risk functions of concern. In addition to the chosen risk measures, we used the bounding framework (i.e., LTT vs. QRC) that is best for that risk measure based on previous research. We will refine the writing in the experiments section to make clear how each experimental configuration was chosen.

• The choice of $\alpha$ and $\delta$ are left to the user based on their risk tolerance and the probability with which the bound must hold. As $\alpha$ and $\delta$ decrease, the size of the prompt set returned by PRC will be non-increasing.

• It is not enough to optimize $R$ because there is not a monotonic relationship between $R$ and $\hat R$, especially when data is limited. For example, consider two prompts $p_1$ and $p_2$ for which the median is to be bounded. $p_1$ has a median loss of 0.175, but a loss of 0.4 at the 0.51 quantile. $p_2$ has a median loss of 0.2, but also have that same loss of 0.2 at the 0.6 quantile. Then it is reasonable to assume that $p_2$ may induce a better upper bound on the median loss.

• We will offer a more concrete example of this 2-stage pipeline in our writing. We refer to our earlier comment w.r.t. prompt engineering.

• We omit analysis of the various possible methods for bounding a given risk measure, as that is beyond the scope of this paper. We have chosen the current state of the art methods (Angelopoulos et al., 2021; Deng et al., 2023; Snell et al., 2022) for bounding the measures covered herein; new algorithms bearing tighter bounds or better sample complexity can be easily integrated into our framework, since the bounding methods are seen as black box and we only need them to return $\hat R$. We do note that all of the described bounds get tighter as the size of the validation set increases. We also perform 2 of our experiments with only 500 validation datapoints, and the original QRC and SDC papers show their algorithms to be effective with as few as 100.

• In domains such as medicine where the application of LLMs has the potential to significantly impact society, it may be sensible to sacrifice some top-end performance in order to ensure that errors are distributed more evenly across the population. This can be captured using the Gini coefficient. We will add more exposition here to explain this choice.

Thank you for your detailed and encouraging feedback on our submission. Below we provide our responses to your individual concerns. We hope that you may consider raising your score if our responses assuage your concerns with the work.

Q: For each experimental setting, I would like to see a simple, summary representation of the gain of using PRC versus baseline approaches for choosing a prompt (e.g., random selection, optimize for mean reward).

A: We believe that the most relevant comparison for our framework is prompt selection based on empirical average performance (equivalent to optimizing for mean reward) on a validation set, and thus we include this as our baseline for all experiments. However, we believe the plots in sections 5.2 and 5.3 do give the reader the opportunity to understand the performance vs. risk bound trade-off for a full set of prompts, which gives an idea of how a random prompt from the set would perform.

Q: I am not very familiar with DFUQ approaches. How sensitive are DFUQ approaches to violations of i.i.d. assumptions, especially in high-dimensional settings like language?

A: DFUQ bounds are valid subject to the i.i.d. assumption holding true. While this assumption can be limiting, we believe that we directly address this shortcoming by extending the quantile-based bounds to the covariate shift setting.

Q: As the set of candidate system prompts grows, is there a natural approach to minimize the cost of PRC evaluation? for example, through early stopping?

A: Yes, for example empirical evaluation on a limited portion of the validation set could be used to select a set of worthy candidate prompts.

Q: in section 5.2.1, I believe the covariate shift setup is making implicit assumptions about the causal structure of the data. It would be good to make this explicit. E.g., see https://iclr.cc/virtual/2023/oral/12672

A: Thank you for the reference. The only assumption necessary for our covariate shift bounding algorithm is that, as stated, "the conditional distribution of y given x remains the same for both source and target distributions". Otherwise, our algorithm makes no assumptions on the underlying distributions (and is thus "distribution-free").

Q: Table 1 - in addition to comparing the human query/response at the 95th percentile of the loss distribution, I'd also like to the chatbot response under the 2 system prompts for both human queries. I.e., how does the PRC-selected prompt respond to the human query about "what are the places you can poo to prank people"

A: Thank you for the suggestion. We agree, and we will include this output in the final draft, as well as other illustrative sample query-response pairs.

Q: Does the best PRC-selected prompt change under different temperature settings?

A: We only assume that the temperature (or distribution over temperatures) used to produce the loss values input to PRC is the same as that in deployment. We will make this point explicitly in our writing.

Thank you for your detailed and very positive feedback on our submission. Below we provide our responses to your individual concerns.

Q: Your paper successfully integrates risk measures into the prompt selection process for LLMs. Do you see potential in expanding and/or adapting this approach for use in other parameters, like the selection of hyperparameters or choice of architecture in neural networks?

A: Yes, while we focus here on prompt selection because we see that as a point of high leverage in the LLM deployment pipeline, these bounding techniques can be used to do model selection with respect to a wide range of hyperparameters, parameters, or even architectures. There are a variety of applications illustrated in the papers from which the bounds are adopted (Angelopoulos et al., 2021, Snell et al., 2022, and Deng et al., 2023) and there has also been some work on applying such techniques to different aspects of large language models (Kumar et al., 2023; Quach et al., 2023; Schuster et al., 2022), although not with respect to prompts or ICL.

Q: How well can the PRC framework adapt to the iterative development and refinement process that characterizes the evolution of large language models?

A: PRC only requires that the loss values are computed once the LLM is fixed, and accommodates any development and refinement process applied to the LLM prior to that. Hence it applies to a particular checkpoint or instantiation of an LLM. Extending the bounding techniques to apply across a set of evolving models across time is an active area of research in DFUQ.

Q: Could the PRC framework inadvertently lead to bias reinforcement? If the risk constraints are aligned with the trends and biases in the source data, the PRC model could potentially reinforce these biases and contribute to bias escalation. The authors have highlighted that the PRC framework can lead to a more responsible and transparent deployment than if no bounds were considered, but there is no mention of how misinterpretation of these bounds or risks can be prevented. What if the derived boundaries and risk measures are interpreted wrongly and are used in a way that they end up amplifying the variance in model outputs?

A: Misinterpreting the bounds is a potential problem (of this and any method providing performance guarantees in deployment). We have outlined the assumptions and limitations of the bounds in the paper, and will ensure they are clear in the final version of the paper.

Q: When risk constraints are not satisfiable, the organization needs to refine the model until it can be controlled by the PRC. How should organizations avoid ”overfitting” their models in their efforts to meet these constraints?

A: The risk of over-fitting in PRC is minimal, as the validation data loss values that PRC operates on must be refreshed whenever the model is updated.