Large language model validity via enhanced conformal prediction methods

We thank for the reviewer for their constructive criticism, and we hope that this rebuttal can both address some of the limitations that they identified and certain misunderstandings.

First, we would like to briefly note that the level-adaptive CP work is novel both in and outside of the LLM setting. In the more canonical regression setting, no previous work shows how to adapt the conformal guarantee at test-time to ensure moderate set sizes.

1. We respectfully disagree with the assertion that the paper starts off unfairly in Figure 1. The intent of that figure is to show that our method identifies the appropriate confidence level at test-time for preserving most of the output's claims. To further clarify, our method issues a guarantee that is different for each test output. While it is true that running the conformal factuality method at 80% would result in fewer claims being filtered for this example, this value would not work well universally over all test examples, e.g. other test examples may require 70% or 90% levels. Thus, the shortcoming of previous methods identified in this figure cannot be corrected by simply running those methods at a different level. One of the main contributions of our work is to develop a method that uses a data-driven approach to learn a desirable level for each individual test point and then issues a filtered response that is well-calibrated to the learned level. We apologize for any confusion caused by this figure. In our revision we will modify the text following Figure 1 in order to provide more context for why we believe this is an appropriate comparison.

2. We believe there may be some confusion here about the statements of our results. The theorems stated in our work are finite-sample exact guarantees that match the style and assumptions of those given in the conformal literature. For instance, note that while our result is more general, if $f$ is a constant function, the stated guarantee in Theorem 3.1 is identical to the typical coverage guarantee of split conformal prediction. We will aim to update the text surrounding our results to clarify this fact. We are unfamiliar with any non-conformal baselines for filtering claims from an LLM output with statistical guarantees (finite-sample valid or otherwise), but if any of the reviewers are aware of alternative approaches that we have not considered we would be happy to compare against them.

3. We agree that we ought to clarify our unconventional error guarantee. While we understand the reviewer's concerns, we chose a non-zero error guarantee because it is impossible to preserve most claims with this guarantee without reducing the level (value of $1-\alpha$) well below $50$% (see Figure 3 in the rebuttal attachment). On the other hand, for $3$ errors, the level function we end up with outputs values that are similar to what we might typically associate with ``high probability guarantees'', i.e. $1-\alpha(X)$ is in the range of 80%-95%. We agree that the utility of a $3$ errors or fewer guarantee is less obvious than having $0$ errors, but if the probability of full correctness is undesirably low (maybe the user does not wish to issue a 40% guarantee in order to preserve most claims), this trade-off may be worthwhile. Moreover, the intent of this experiment is to demonstrate the flexibility of our method to accommodate weaker error guarantees compared to exact factuality, not to suggest that $3$ errors is the desirable outcome for all applications. Overall, we agree with the reviewer that this choice requires further explanation in the main text, and we will revise the manuscript accordingly.

4. We agree that the exchangeability assumption is a limitation. We are not certain what the reviewer means by a given type of data, but in general we believe that the limitations of our method are no more significant than any existing approach to this problem. Additionally, as our methods target a form of conditional validity, they will provide robustness to some settings beyond exchangeability, e.g., when the test data undergoes a covariate shift.

5. Many of the reviewers inquired about additional experiments and we agree that the single real-data example given in the manuscript is in need of additional support. We have performed additional experiments that include the requested ablations as well as a large second experiment on a more real-world dataset involving medical question-answering. These results are shown in our global rebuttal to all reviewers and we hope that will assuage the reviewer's concerns.

We thank the reviewer for their positive comments and feedback. We address specific concerns below:

1. We agree that it is costly to obtain these tuples and it may not be feasible to do so in all cases. Nevertheless, given the large resources invested in LLM development by corporate labs, we do not believe that it is unrealistic that human-labels could be obtained for many real-world datasets of interest. Additionally, we hope that our work will help to provide a useful framework for further development in this field, and we have identified at least a few generation tasks where this methodology may prove helpful.

2. We are not certain which metrics the reviewer is referring to. In our revision, we will aim to ensure that the notions of calibration and retention utilized in our work are clearly defined.

3. The validity guarantee output by our method holds in expectation over the calibration set regardless of its size. However, it will be more variable conditional on the calibration set if the dataset is smaller or relevant subsets are underrepresented. In general, this method will perform best if the complexity of the factuality guarantee (i.e. the complexity of the function class $\mathcal{F}$) is not too large relative to the size of the calibration set. This leads to a model selection question that is analogues to what one faces in many standard regression problems. A detailed discussion of these issues can be found in Gibbs, Cherian, and Candes (2023). Since this article is primarily concerned with novel improvements to the methodology of Gibbs, Cherian, and Candes (2023) that enable its application to LLMs, we do not discuss these selection details in much depth here.

4. The Wikipedia dataset might appear synthetic since most people are not interested in writing biographies. Nevertheless, we believe it is still representative of the types of tasks LLMs may be used for. To help further demonstrate the range of possible applications of our method, we have performed additional experiments on a second medical question-answering dataset. We believe this example constitutes a highly relevant and realistic application for chat bots. Figures showing results from this experiment can be found in our global rebuttal to all reviewers. Overall, we find that our methods perform well on this example, obtaining both good calibration and improved claim retention.

We thank the reviewer for their feedback. We will look to prepare a revision that clarifies our experimental choices and provides additional experiments addressing the reviewer's concerns. We hope the following responses address the reviewer's concerns.

Weaknesses:

1. We agree that this would be helpful, and we performed additional experiments with the requested ablations for both the Wikipedia and new medical question-answering experiment. Discussion of these results as well as associated figures can be found in our global rebuttal to all reviewers.

   In our experiments, the frequency scoring method is most effective, and the improvements obtained by our method are robust to changes in the score. In our revision, we will prepare additional figures reproducing our results for alternative sub-claim scoring approaches.

2. The difference between the two methods in Figures 3, 4, and 7 may be less pronounced if a superior scoring function was selected. The goal of this experiment is not to show a comprehensive comparison, but instead to demonstrate the theoretical pitfalls of marginal boosting (i.e., conformal training). In general, if the scoring function is perfectly chosen, no boosting method (targeting marginal or conditional validity) substantially alters the performance of the method. This is also apparent in the Wikipedia example where the frequency scoring function far outperforms the other approaches and boosting simply recovers the performance of the best scoring approach. We believe that in many realistic scenarios, users do not have a priori access to a strong scoring function and thus, as this example demonstrates, conditional boosting can offer significant benefits over marginal boosting. This can be partially seen in our results on the Wikipedia and medical question-answering datasets where boosting produces a stronger ensemble of many weak scoring functions.

3. We apologize for the lack of clarity in the experimental set-up. We will aim to prepare a revision that more clearly organizes this information.

Questions:

1. The results we prove hold in expectation over the calibration set. We agree that the calibration-conditional validity of the filtering method will be more variable if the calibration set is smaller (or similarly if there are fewer samples in some subgroups). We will add additional information about the sample sizes to the figures in our revision.

2. The primary reason we used the Wikipedia/FActscore dataset was because of the availability of high-quality synthetic annotations. Mohri and Hashimoto hand-label the correctness of each line of the response for $50$ examples. Unfortunately, running our method with any complex conditional guarantee on a such a small calibration set would lead to highly variable outcomes. Thus, we have chosen to generate synthetic ground-truth using GPT-4 with the original Wikipedia page passed in as additional context. Prior literature has validated that this method leads to an accurate factuality annotation for the FActscore benchmark. By contrast, we are unaware of any existing methods for obtaining synthetic ground truth on the other datasets considered by Mohri and Hashimoto. Nevertheless, we do agree that additional experimental validation would be helpful. As a result, we have also analyzed a medical long-form question answering dataset (released just prior to the submission deadline) where both expert and high-quality synthetic annotations have been made available.

3. We thank the reviewer for identifying this typo and will correct it.

I thank the authors for addressing the questions and for sharing additional experiments. I look forward to the revision with suggested changes in future versions. I remain positive about the work and would like to keep my score as above.

We thank the reviewer for their positive comments and constructive feedback. In response to the reviewer's specific comments:

1. The proof of this result is given in Appendix A. Our assumption that $L(\emptyset, \cdot) = 0$ is necessary to ensure the score function, $S(\mathbf{C}_i, \mathbf{W}_i)$ given on line 191 is well-defined. On the other hand, our assumption that the loss is monotone in the filter is necessary to derive an equivalence between loss control and the dual variable whose distribution is analyzed in the proof of Theorem A.1. As stated in Appendix A.1, Theorem 3.1 then follows as a corollary of Theorem A.1. In particular, note that on Line 423 in the proof of Theorem A.1 we explicitly utilize our monotonicity assumption. In our revision we will add additional clarification to the main text of the manuscript discussing why these assumptions are necessary.

2. The "Conformal Risk Control" article of Angelopoulos et al. (2022) is referenced in our discussion of related work (see line 32). The approach taken to quantifying the uncertainty of LLM outputs taken in Angelopoulos et al. (2022) is quite different from our work and so we have not given an in-depth comparison. Nevertheless, we agree that the guarantees and high-level approach taken there are similar to our work and we will look to clarify this connection further in our revision.

3. We have referenced Quach et al. (2024) in our discussion of related work (see line 32). We thank the reviewer for bringing Kang et al. to our attention and will add this article to our discussion of related work in our revision. Like Quach et al., Kang et al. analyzes how many LLM generations are required to control a monotone risk. While the conformal methods used are related, we would emphasize that our approaches are both quantitatively and qualitatively different, i.e., we filter a single generation to obtain high-probability control of the loss.

4. Our goal is to adapt the coverage level of our method to the underlying difficulty of filtering falsehoods out of each example. To do this, we fit an adaptive function, $\alpha(X)$ in order to ensure that running our conformal method at level $1-\alpha(X_{n+1})$ will preserve at least a user-specified fraction of the LLM response. Then, we run an augmented quantile regression-type procedure in which data point $i$ is assigned loss $\ell_{\alpha(X_i)}(S(\mathbf{C}_i,\mathbf{W}_i) - g(X_i))$.

5. We are not entirely sure what exact alternative method the reviewer has in mind. Perhaps the reviewer is asking why we cannot simply use the value $\alpha^*(X_{n+1})$ such that when we run split conformal at this exact level, the desired percentage of claims are retained. Alternatively, perhaps the reviewer is interested in the method where we use $\alpha(X_{n+1})$ as a plug-in everywhere so that data point $i$ receives loss $\ell_{\alpha(X_{n+1})}(S(\mathbf{C}_i,\mathbf{W}_i) - g(X_i))$. Unfortunately, both these methods can incur selection bias and neither is valid in general. On a technical level, both of these approaches treat the test data point asymmetrically to the training data. This breaks the exchangeability of the data and thus does not prove a validity guarantee. On the other hand, our method, which indeed modifies the pinball loss, maintains exchangeability of the data and thus guarantees control of the loss. This result is rigorously proven in Theorem A.1.

6. The conformal training method in Stutz et al. (2021) backpropagates through the split conformal cutoff. Since the split conformal cutoff is defined to be a quantile of the calibration set, the value of cutoff is actually a particular calibration set score, i.e., $\text{cutoff} = S_{\lceil (n + 1) (1 - \alpha) \rceil}$. Differentiating with respect to some $\lambda$ that parameterizes the scoring function is then relatively straightforward as one may simply examine what happens to the score $S_{\lceil (n + 1) (1 - \alpha) \rceil}$ in isolation. It is worth noting that Stutz et al. (2021) use a "differentiable sorting" algorithm to obtain smoother gradients. By contrast, our cutoff is defined by the function of a solution to an optimization problem. The derivative of such an optimum is typically obtained via a second-order Taylor approximation (cf. influence functions). However, in our case the objective of the optimization problem is not differentiable and thus we cannot apply such an approximation. As we show in Proposition 3.1, this technical challenge can be overcome; despite the non-differentiability, we prove that the optimum remains locally linear in the parameter of interest.

I would like to thank the authors for the clarifications.

My concerns 1,3,4,5 are addressed, but for Q2 and Q6, I appreciate additional clarifications:

Q2: In conformal risk control paper, the conformal guarantee is valid for any risk function and any parameterized generation algorithm. Therefore, the method can be exactly tailored for the problem here. Concretely, the algorithm can be exactly cutting off claims with low scores with the parameter as the threshold. By running conformal risk control, we can identify the sets of parameters (i.e., thresholds for cutting off) which achieve valid hallucination risk. In this sense, the contribution in sec 3.1 is not the first trial in the conformal literature and I would like that the authors clarify more on the novelties for Sec 3.1.

Q6: I do not understand why the objective leads to additional challenges. The difference that paper made compared to standard conformal prediction is to use another adaptive loss function as the conformity scores. Why this will lead to a non-differentiability challenge? In particular, why is this not resolved with differentiable sorting as conformal training paper?