Document Summarization with Conformal Importance Guarantees

Thank you for your efforts to review our work! It is great to see that you found our perspective on summarization novel. Below, we will address the weaknesses you brought up.

W1:

The paper contains a large number of formulas … which negatively affects readability. The authors are encouraged to summarize their main contribution to conformal prediction or extractive summarization. … is this paper fundamentally about applying conformal prediction to the extractive summarization task …?

It is possible that there are some misunderstandings in your review about how we brought together conformal prediction and document summarization. Our contribution is not to apply standard conformal prediction to summarization tasks, but to apply the techniques behind conformal prediction to summarization tasks. Our settings and results (Theorem 1) are entirely different from that of conformal prediction as you may have seen it applied to classification or regression tasks. Thus formulas and proofs are needed to make it rigorous. Conformal factuality [41] is another example of applying the techniques of conformal prediction to a novel LLM setting (question-answering), instead of applying standard conformal prediction. For that reason, [41] also provides proofs and formulas.

In summary:

• conformal prediction [57,54, 2] aims for a coverage guarantee that the ground truth answer is contained in the prediction set (Equation 1);
• conformal factuality [41] aims for a coverage guarantee that all retained claims are factual (Equation 3);
• conformal summarization (our work) aims for a coverage guarantee that all (or a fixed percentage of) the important sentences are retained (Equation 4).

While there are common ideas behind all these approaches, the settings and the essential coverage guarantees are notably different, and each requires independent proof that coverage is obtained.

W3:

The paper lacks reporting on a primary metric in conformal prediction: Set Size, which quantifies prediction uncertainty.

Following the above explanation, since conformal summarization is not the same as conformal prediction, we did not report “set size” directly. Set size is indeed a primary metric for conformal prediction and quantifies uncertainty. In a simple classification problem, for example, there is a fixed set of labels to choose from for every datapoint, and the conformal set is just a subset of all possible labels. For the same level of coverage $1-\alpha$, smaller sets are more useful (L72) which makes average set size a good metric.

For our problem of document summarization the label set is no longer fixed. Instead, each summary contains a subset of all sentences from the long-texts $x$, which vary in length. The number of sentences retained in the summary is still a good measure of quality, as we want the shortest summaries that retain all (or a fraction $\beta$) of the ground-truth important sentences (L111). It also still quantifies uncertainty, as longer summaries indicate the model is more uncertain about what content is important. But, since the total number of sentences in $x$ varies, rather than use the count of sentences in the summary (similar to set size in conformal prediction setting), it makes more sense to normalize the “set size” as the fraction of sentences removed (L233). This is the measure of “conformal set size” we report in Table 2 and other figures.

W2:

The motivation behind this work is not clearly articulated. Given that the method is restricted to extractive summarization, is this still meaningful in the era of large language models (LLMs)?

Instruction-tuned LLMs are powerful tools and natively can perform abstractive summarization with basic prompts. One might wonder why we bother with extractive summarization at all. We comment on this in the paragraph on L25. The main motivation is that prompted LLMs do not provide any guarantees that they will extract the information a user deems important. Even worse, they are well-known to sometimes hallucinate information beyond what was provided, or make logical errors in interpreting that information. For example, a study using GPT-4 to summarize emergency department visits found hallucinations in 42% of summaries and that 47% omitted clinically important information [A]. Many similar evaluations of LLMs are available in the literature. In response to Reviewer fdXq (Q1) we tested how well GPT-4o mini retains ground-truth important information with naive abstractive summarization. The results show only 26% of summaries had recall over 80% (low coverage), and overall recall was only 52% (low recall).

By using extractive summarization it is impossible to fabricate information, and by using conformal prediction we can align the extracted information with a user’s preferences (via the labeled calibration set) and give statistical guarantees on recall and coverage.

Let’s consider an analogy to the popular task of question-answering. Many production systems avoid “pure” LLM generation and instead use Retrieval-Augmented Generation (RAG) [B], which grounds generative outputs in retrieved text to mitigate hallucinations while preserving fluency. RAG is a two-step process that first retrieves from a document corpus and then generates from the retrieved documents. The first step involves finding the most relevant document to the question, in which the output format is controlled (not in the form of free text). By design there cannot be hallucination in the retrieval step, since it only accesses a pre-defined corpus. The second step, generation, has its output as free-form text, i.e. uncontrolled. Thus it could contain hallucinations. But as the inputs to this step are controlled, RAG greatly reduces hallucinations on this step.

Our framework for summarization addresses the same reliability concerns as RAG, and is less prone to hallucination than “pure” LLM summarization. Our method can be thought of similarly to the retrieval step in RAG, in that the output is controlled -- the output sentences are directly from the original text. Just like the retrieval in RAG, we score and rank content, but go further and provide coverage guarantees. The output of our approach can be fed into a second stage of abstractive summarization using an LLM, similar to the generation step in RAG, which serves to improve fluency and further shorten the text. Thus our approach and abstractive LLM summarization are complementary. We provide example experiments in response to Reviewer fdXq (Q3) that show using our conformal summarization as a pre-processing step improves the recall and coverage of abstractive summarization.

[A] Williams et al. “Evaluating large language models for drafting emergency department encounter summaries” PLOS Digital Health 2025

[B] Lewis et al. “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks” NeurIPS 2020

W4:

Instead of defining the threshold ​q_hat using a preset α, an alternative approach would be to use p-values, as suggested by Vovk et al. (2005). It would be interesting to evaluate the performance of the proposed framework using the p-value method for constructing prediction sets.

This is another interesting question, and luckily there is well understood theory around it. Essentially, the viewpoints of conformal prediction via coverage (modern view [2], which we used) and p-values (classic view [57]) are equivalent and interchangeable. They provide a different statistical interpretation, but they produce the same results. Hence, there would be no difference to the performance of our framework if we were to use the p-value method for constructing prediction sets.

Let’s take the classification setting again for example. In the modern view of CP for coverage guarantees we want to construct a prediction set that contains the true label with high probability $1-\alpha$. The interpretation here is that the sets quantify the uncertainty of the underlying predictive model, with larger sets indicating more uncertainty. Mechanically, we compute a conformal score for each possible label and include it in the set if the score is greater than a calibrated threshold, which is a quantile of scores on a calibration set. The quantile to use depends on the desired coverage $1-\alpha$.

In the classic view using p-values we want to construct a prediction set of all labels that are statistically consistent with observations from the calibration set. The interpretation is that CP conducts multiple hypothesis tests and controls Type I errors. Mechanically, for each possible label we compute a p-value representing what fraction of examples in the calibration set are as non-conforming as the proposed label. Then we include all labels where the p-value is greater than a significance level $\alpha$.

The definitions of the quantile and p-value in the two views, along with their set construction method, can be exactly mapped onto one another making them totally equivalent in practice.

Thank you for your time spent reviewing our paper, and for noting that our experiments were comprehensive. We are happy to respond to your questions and the weaknesses you raised below.

W1:

The proposed approach might not scale to longer documents as it requires per-sentence judgment of importance using LLMs.

We acknowledged in L322 that the datasets we used were limited in the length of documents they contained, however there is no fundamental limitation to applying our method on longer documents, only increased cost to score sentences. As such, we have repeated our experimental setup behind Table 2 on longer documents from the SciTLDR dataset [9]. We originally used the Abstract-Introduction-Conclusion subset of SciTLDR, with the average input document length being ~41. Here we use the full-document subset of SciTLDR, where the entire paper is used as the input document, increasing the input length to an average of 216 sentences, with resulting performance below:

For the longer documents the positive label rate is much lower (1.4%), and so we see the method is able to remove a higher fraction of sentences, while still maintaining a low rate $\alpha=0.2$ of failure to achieve high recall $\beta=0.8$.

While we used sentences as a fundamental unit of text to perform conformal summarization, this was a choice for convenience; the underlying method is flexible for any defined span of text like paragraphs or strings of $N$ words. Simply break the long-text into spans, score their importance, and filter out spans with scores lower than the calibrated threshold. This is one way to scale to very long documents if per-sentence judgements become prohibitively expensive - use paragraph-sized spans to reduce the number of spans that need to be scored.

W2:

The effects of the calibration data is not explored. It would be informative to have some measure of performance of top-performing importance model as a function of the calibration dataset size.

The effect of calibration dataset size $n$ in conformal prediction is well understood theoretically, and in summary has no effect on the summary conciseness on average. The coverage guarantee of conformal prediction is famously “valid in finite samples” which means that it holds statistically for any finite calibration dataset. In practice, $n$ controls the variance of the coverage viewed as a random variable over the calibration data.

CP assumes the calibration data is drawn from a distribution, and that test datapoints are exchangeable with the calibration data. The conformal threshold $\hat q$ is a random variable that depends on the calibration data; drawing $n$ new calibration points and rerunning CP would give a different value for $\hat q$. In turn, the conformal prediction sets would also differ, leading to slight differences in empirical coverage and conciseness. The main theorems of CP say that the coverage, as a random variable over the calibration data, will have an average that is no less than $1-\alpha$ (e.g our Theorem 1, or Equation 1). However, we know more about the distribution than just its average. For typical applications of CP, it follows a Beta distribution with a variance that scales like $O(n^{-1/2})$. For a textbook-style explanation of these details, see Sec 3.2 of [2].

So, while slightly different conformal thresholds will in practice give different recall and conciseness for summaries, the fluctuations are random and are not expected to give better or worse conciseness as $n$ increases. $n$ controls the size of the random fluctuations.

The only other notable effect of $n$ is on the tightness of the upper bound in our Theorem 1 (Compare to Equation 1 from [2]).

Recognizing the costly nature of obtaining labels, we limited our experiments to $n=100$ and still obtained strong coverage results demonstrating that very little labeled data is needed in practice.

W3:

The Appendix is missing…

The appendix is provided as Supplementary Material rather than added to the main PDF, which were both options given in the PC’s instructions this year. Please download the zip file from OpenReview, and we can answer any further questions it brings up for you.

Q1:

In Table 2, it would be helpful to have a baseline line vanilla LexRank, subject to a summary budget in words matching the average words from conformal importance summaries.

In the paper we used LexRank as a scoring function for conformal summarization, and based on your suggestion have now run a baseline using LexRank directly for summarization on the ECT dataset, using a word budget that matches the average from the other conformal methods. For reference, this ranges from 834 words (Gemini 2.5 Flash) to 1101 words (Cosine Similarity Centrality).

We tested LexRank in both continuous mode and discrete modes. For the discrete mode, we used the four edge-weight thresholds (0.1, 0.2, 0.3, 0.4) as done in [20]. Additionally, we experimented with two initialization strategies for LexRank’s TF-IDF-based embeddings: one using only the calibration dataset, and another using the combined calibration and test datasets. We only report the latter as it is what we used in the main paper and we found the difference in results to be negligible.

We evaluate the recall of important sentences under this framework, the results of which can be found in the table below. Columns indicate which alternative conformal method was used to derive the average text length to use as the budget for LexRank.

Recall varies between 0.69 (0.1 edge weight threshold, based on Gemini 2.5 summaries) up to 0.87 (0.4 edge weight threshold, GUSUM summaries). By comparison, all our conformal methods from Table 2 achieve recall $\geq 0.80$ for 80% of the datapoints. Hence, conformal summarization with the best LLMs (Gemini or GPT models) tends to outperform LexRank at retaining important information for the same length budget.

This is also consistent with our AUPRC measurement, which is purely a measure of how well an NLP method can discriminate between important and unimportant information (i.e. it does not rely on conformal methods at all). From Table 2, LexRank has AUPRC of 0.22 on ECT, significantly worse than Gemini or GPT models (0.30 - 0.43 on ECT), but still roughly comparable to other classical NLP techniques.

The fact that base LexRank’s recall is not controllable, unlike the conformal methods to which we compare it, also shows the utility of conformal prediction in the summarization setting. Integrating LexRank with conformal summarization removes this limitation of the base method alone.

Thank you for clarifying your concerns with W1 and W2. For W1, we refrained from claiming in the paper that the proposed method works well on spans other than sentences. We agree that this would need to be experimentally investigated before such claims are made. Still, we hope we have addressed your original concern with the new results on larger documents which show that sentence-level scoring can still work at the scale of hundreds of sentences.

For W2 on the importance of calibration set size, we can easily explore the coverage and its variance for even smaller calibration sets to cover the case where human-labeled examples are very expensive. Below, we show results similar to Figure 2 where we track the empirical coverage for various target coverage levels ($1-\alpha$). We now ablate over calibration set sizes with values ranging from 25 to 100 samples, and present both the mean and standard deviation of the empirical coverage random variable. Since 100 samples were originally provisioned for the calibration set, we simply subsample them at random for the ablations with fewer samples. The rest of the experimental settings match those of Figure 2, with $\beta=0.8$ and Gemini 2.5 Flash.

Mean of Empirical Coverage

Std of Empirical Coverage

As guaranteed by our Theorem 1, the mean empirical coverage is consistently above the target minimum coverage $1-\alpha$, and below the upper bound $1-\alpha + \frac{1}{1-n}$, which is 1 to 4 percentage points higher based on $n$.

More importantly for your question, we find the variance decreases as the calibration set size increases, which is predicted by the theory we referenced previously. At $n=50$ which you asked about, the standard deviation of empirical coverage is 1 to 2 percentage points higher than $n=100$. However, the mean is also higher by 1 to 2 percentage points, meaning that the “failure cases”, where the empirical coverage falls below the target level due to random fluctuations, have about the same severity.

You originally asked about the performance of the methods as a function of set size. Below we show the main metric for performance -- fraction of sentences removed -- for the same settings as just discussed with $1-\alpha=0.8$. We have rerun the method for 20 random calibration/test splits and present the mean and standard deviation.

Fraction of Sentences Removed:

As $n$ decreases there is a slight decrease in metric values which corresponds to the higher average coverage we found above. Overall, there is not expected to be a strong dependence of performance on calibration set size from the theory of conformal prediction. The mechanism of CP tries to estimate the value of the conformal threshold which will lead to a desired coverage level. The “true” value of the threshold that will achieve a target coverage $1-\alpha$ is the same for all calibration set sizes, and changing the number of calibration points only affects the variance of the estimated threshold. In turn, the selected threshold directly determines the method’s performance regardless of $n$.

If you find these points valuable, using the additional content page for the camera-ready version we can include a complete discussion, covering all conformal scoring functions.

Thank you for your positive review! We are pleased to see that you found our theory “solid”, and will address your questions and comments below.

W1:

The method depends on having a predefined set of important sentences … can be subjective or expensive to collect. The quality of the summary’s guarantee is only as good as the quality of these labels. There could be some more discussion on how robust this method is to noisy or imperfect importance labels.

We agree with your argument that in real-world, professional, and sensitive domains, sentence level importance labels would be hard/expensive to collect. While modern LLMs have been shown to achieve good performance on many out-of-domain scenarios with zero/few-shot inference, it is known that their performance could drop on niche and/or sensitive domains. It should be recognized that conformal prediction does not require many labels compared to other ML-based approaches. We used only 100 labeled samples in our calibration sets and achieved tight coverage at high recalls (Fig 2).

Other than the direct and expensive approach of hiring professionals to label training samples, one possible idea is to adapt domain-finetuned LLMs for the purpose of either scoring or labeling. There has been a significant amount of work demonstrating that adapted LLMs show increases in accuracy, reduction in hallucination, and/or more alignment with expert preferences (e.g. [A,B]). All these indicate that they would serve better as both a scoring function as well as a labeling mechanism.

There has been research around conformal prediction’s robustness toward label noise [C], as well as how the conformal process could be adapted to explicitly handle label noise [D]. Since we use the same theoretical principles as these works, their conclusions may extend to conformal summarization, although it is not the focus of our present paper.

[A] Woo et al. “Synthetic data distillation enables the extraction of clinical information at scale”, npj Digital Medicine, 2025

[B] Fatemi et al. “A Comparative Analysis of Instruction Fine-Tuning LLMs for Financial Text Classification”, arxiv:2411.02476

[C] Einbinder et al. “Label Noise Robustness of Conformal Prediction”, Journal of Machine Learning Research 2024

[D] Penso et al. “Estimating the Conformal Prediction Threshold from Noisy Labels”, arXiv:2501.12749

W2:

While the motivation is to improve summarization in high-stakes domains like health and law, evaluation is only run on standard summarization benchmarks (news, scientific papers, dialogues). In a truly high-stakes setting, importance could be multifaceted and harder to capture with a single proxy.

One of the five datasets we used did cover the domain of healthcare (MTS-Dialog). We agree that importance can be multifaceted. We discussed in and after L168 how our method can accommodate differing viewpoints on what is important through user-specific calibration set labels and importance score functions. In this way, each user can receive tailored summaries with coverage guarantees that correspond to their personal definition of importance.

W3:

The method is heavily focused on coverage guarantees, and does not consider other aspects of summary quality. While completeness (high recall) is critical, things like coherence or usefulness are not being evaluated.

Given that coverage is satisfied, the main metric that demonstrates the effectiveness of our method is the conciseness of the summary -- how much non-important information can be filtered out without sacrificing recall of important information. Since we are focusing on extractive summarization, the notions of coherence and usefulness are less relevant. There is no expectation that an extractive summary preserves coherent paragraph structure and flow, while usefulness is solely dictated by the recall of important information. Below in the response to Q3 we discuss extending our extractive method with abstractive post-processing, where your suggested metrics become more relevant.

Q1:

Although existing abstractive summarizers rarely reproduce the exact sentences labeled important, they often paraphrase them well enough that the underlying facts are still conveyed. Measuring how much of this critical content they actually cover (e.g., by computing the recall of important sentences using semantic-entailment or similarity rather than verbatim overlap) ­would clarify how frequently those baselines meet a chosen recall target \beta, even without any formal guarantees.

This is a great suggestion! We tested GPT-4o mini on ECTSum (2,322 test samples), prompting it to produce abstractive summaries directly from the original source text, with the target of retaining all important content ($\beta = 1$). While Table 2 reports results for our conformal methods at $\beta = 0.8$, here we used $\beta = 1$ to measure the LLM’s raw ability to preserve information. For evaluation we used semantic-entailment to determine recall; each ground-truth important sentence was compared to the generated summary using an LLM-based evaluator to check if the information was retained in the summary.

For this direct, single-stage abstractive summarization approach we found:

• Coverage: Only 26% of abstractive summaries had recall $\beta \geq 0.8$, compared to $1-\alpha=0.8$ attained with our conformal methods.
• Recall: Average recall was 52% (std=35%), far below the instructions for $\beta=1.0$.
• Length: For the summaries that retained a fraction $\beta\geq 0.8$, the mean fraction of sentences removed was 65% (std=25%), which was higher than the best conformal method at 37% (Table 2).

These results show that naive prompting of an LLM is not likely to correctly retain the ground-truth important information, and clearly does not give any control over recall or coverage. LLMs tend to aggressively shorten content when instructed to summarize.

We ran a similar experiment but with 10 in-context labeled examples added to the prompt. Please see our response to Reviewer sN9c, W3/Q1. This increased the coverage to 40% and recall to 67%, but decreased conciseness to 58%.

Q2:

Given that the motivation is centered on critical-domain summaries (medical, legal, etc.), have you considered how to validate CIG in such settings? What should such expert-defined important content look like, and what would be the obstacles and ways to overcome them?

We did test our method on one medical dataset (MTS-Dialog), and this is a great example of what expert-defined importance could look like. The MTS-Dialog consists of doctor-patient conversations. It is reasonable to expect that in a real-world setting like this, both the doctor and patient may want a summary of their conversation, but the material each person considers important could be very different. A doctor may focus more on the symptoms described by the patient, while a patient would highlight the doctor’s medical recommendations as most important. For the same set of conversations, we may have a set of ground truth labels from doctors, and another from patients, each of which would lead to different conformal thresholds and hence prediction sets.

Going further, the expert definitions of importance could be incorporated into the importance score function. We found that LLM scoring with a basic prompt performed very well across datasets (Table 2). However, incorporating more information about what the expert deems important could be done by providing few-shot examples labeled by the expert in the prompt to activate the in-context learning capabilities of LLMs. In this way, the LLM scoring would not simply rely on parametric knowledge of what is important from the LLM’s pre-training data, but would directly incorporate expert knowledge about the dataset and preferences at hand.

Q3:

Have you thought about combining the current approach with abstractive techniques? You could first identify a set of important sentences, then generate a refined summary with an LLM. If you compare summaries generated via this two-stage approach to those generated via single-stage prompting of the same LLM, how do they perform in terms of quality and coverage?

Adding our extractive approach as pre-processing before using abstractive summarization could remove some of the unimportant noise sentences and result in a more focused summary.

We provide experiment results of this two-stage approach: (1) use our conformal summarization to extract a set of sentences meeting a target recall threshold ($\beta=0.8$) with high coverage ($1-\alpha=0.8$), and (2) prompt GPT-4o mini to generate a cohesive abstractive summary from the extracted sentences. Note that we did not instruct the LLM to retain all important information, just to summarize the remaining text. The results on ECTSum (2,322 samples) are below and we compare to the single-stage abstractive summarization from Q1 above:

• Coverage: 42% of abstractive summaries samples reached $\beta \geq 0.8$, compared to 26% without pre-processing.
• Recall: Average recall was 67% (std=31%), compared to 52%.
• Length: For the summaries that retained a fraction $\beta\geq 0.8$, the mean fraction of sentences removed was 57% (std=25%), compared to 65%.

Hence, there does appear to be a benefit in how the LLM is able to identify important information if the pre-processing is used, although summaries were not quite as concise.

For another perspective - testing how well an LLM can preserve the recall and coverage guarantees in a two-stage set up - please see our response to Reviewer sN9c in W1/Q2.

Thank you for reviewing our paper! We are glad to hear you found our use of conformal prediction to be “novel” and will address the weaknesses and questions you raised.

W1/Q2:

The method cannot handle abstractive, long-range, or structurally complex documents… The assumption that important content exists at the sentence level is limited. Please provide examples and analysis on long document summarization tasks.

Abstractive: Our method could be combined with today’s highly performant LLMs to perform abstractive summarization as a post-processing step on our extractive summaries. The extractive summary with coverage guarantees can be fed into an LLM with instructions to further paraphrase and shorten the text, but without removing any important content. As long as the LLM can maintain information and avoid hallucination, it will provide the benefits of abstraction while maintaining our statistical guarantees.

However, in practice we find that LLMs are not good enough yet to maintain all important content while meaningfully reducing summary length further. Still, there can be other benefits of abstractive summarization, such as improving the flow and style of writing which may be broken up when only extracting sentences.

We demonstrate progress in this regard by using our conformal method with a target recall threshold $\beta=0.8$ with $1-\alpha=0.8$ coverage, and then prompt GPT-4o mini to paraphrase and shorten the text while retaining all of the information of the extractive summary. The results on ECTSum (2,322 samples) are summarized below:

• Length: The abstractive post-processing step slightly increased the length (sentence count) compared to the extractive summary by 3% on average, though with very high variance (std=51%).
• Coverage: Overall coverage was 61%, a drop from the 80% achieved with our method alone.
• Recall: Overall recall was mean 75%, std 34%.

So while there may be some stylistic benefits to an abstractive post-processing step, there is no further reduction in length and a loss of some important information.

Note that the evaluation of recall for the abstractive method must be done on a semantic basis and is therefore less precise. For a given datapoint, each ground-truth important sentence was passed to an LLM evaluator along with the generated summary, and the evaluator was prompted to determine if the ground truth information still appeared in the summary.

---

Long-range: We acknowledged in L322 that the datasets we used were limited in the length of documents they contained, but this does not mean that our proposed method fails on longer documents. As a test, we have run our method on a dataset with longer document length: in the paper, we used the Abstract-Introduction-Conclusion subset of SciTLDR, with the average input document length being ~41. Here we use the full-document subset of SciTLDR, where the entire paper is used as the input document, increasing the input length to an average of 216 sentences, with resulting performance below (all experiment settings are as in Table 2):

For the longer documents the positive label rate is much lower (1.4%), and so we see the method is able to remove a higher fraction of sentences, while still maintaining a low rate $\alpha=0.2$ of failure to achieve high recall $\beta=0.8$.

---

Sentence-level: While we used sentences as a fundamental unit of text to perform conformal summarization, this was a choice for convenience; the underlying method is flexible for any defined span of text like paragraphs or strings of $N$ words. Simply break the long-text into spans, score their importance, and filter out spans with scores lower than the calibrated threshold. This point is another way to address summarization of very long documents - use paragraph-sized spans for example to reduce the number of spans that need to be scored.

The assumption is less that important content exists at the sentence level, but more that it can be captured using non-overlapping spans. However, the upside of using this assumption and method is statistical guarantees on coverage and recall, something that cannot be obtained with abstractive techniques (see Q1 below).

W2:

The method performance depends entirely on the quality of the sentence scoring function.

It is natural that any conformal prediction approach relies on the quality of its conformal score function, so we hardly see this as a weakness. For classification problems, ample research has been done on the design of high-quality scoring functions [52, 3, 30], and there is room for future research to build on our framework with new score functions. We proposed (L175ff) and evaluated (Table 2) several score functions to demonstrate the effect of score function design on the overall performance of our method, and found that today’s strong LLMs give the best results. This is encouraging, as our method can easily swap out for new LLMs that are developed over time, likely giving improved summarization performance “for free”.

W3/Q1:

The experiment lacks direct comparison with abstractive methods or GPT-style Models for document summarization. Please provide direct comparison with SOTA LLM-generated summaries.

We have run several more tests using LLMs purely in an abstractive capacity and produced metrics on recall and coverage to compare with our proposed extractive method. We evaluated GPT-4o-mini on ECTSum (2,322 test samples), prompting it with 10 labeled examples from the calibration set to enable in-context learning, with instructions to produce abstractive summaries that retain all important information (target $\beta=1$). Even though in Table 2 our conformal methods targeted $\beta=0.8$, we used $\beta=1$ here to focus the evaluation on the LLM’s ability to capture important information. The evaluation of recall was done in the same way as described above for W1/Q2.

Even with in-context examples, GPT-4o-mini failed to reliably retain all the important information:

• Coverage: Only 40% of abstractive summaries achieved $\beta \geq 0.8$, compared to $1-\alpha=0.8$ attained with our conformal method, underscoring the lack of control over recall with naive LLM prompting.
• Recall: Average recall was 67% (std=31%), far below the instructions for $\beta=1.0$, and the guaranteed recall we achieved with our conformal method (minimum $\beta=0.8$ with high probability).
• Length: For the summaries that retained a fraction $\beta \geq 0.8$ of the important information, the mean fraction of sentences removed was 58% (std=26%), which was higher than the best conformal method at 37% (Table 2).

In summary, abstractive summarization with LLMs, even with labeled in-context examples, tends to aggressively shorten text and results in low recall. The biggest drawback here is that the LLM does not give the user any control over recall or coverage (we prompted for 100% recall and achieved 67%). In contrast, our conformal approach consistently meets user-specified recall and coverage targets.

We ran a similar experiment but without in-context examples for comparison, but this resulted in even more aggressive shortening leading to worse coverage. Please see the response to Reviewer fdXq if you are interested.