Learning to Reject with a Fixed Predictor: Application to Decontextualization

Thank you for your thoughtful feedback. We have carefully addressed all the questions raised. Please find our responses below.

Weaknesses:

1. The experiment setting is not well explained and the settings are not comprehensive. As the authors mentioned, the LLM predictors are their main focus but in the experiments, there is only one type of LLM. As T5X is a family of models, there are other choices and more architectures from other families will be more comprehensive for evaluating the proposed loss.

Response: We kindly ask the reviewer to elaborate on which part of the experiment setting was not sufficiently well explained. While it is true that there generally exist many LLM architectures, it would not be feasible to evaluate all of them. By showing very strong results on T5X XXL, a standard model, we demonstrate that there is at least one architecture for which our surrogate loss is very useful.

2. The loss function is tested on image classification but on a tiny setting and this pure classification task does not really relate to the LLM setting.

Response: Since annotation is expensive for the LLM setting, we chose to provide additional experimental evaluation of our proposed surrogate loss on image classification. For the LLM setting, please see our experiments on decontextualization.

3. To train the rejector still required to label new information, it is hard to distinguish whether the improvement is from the additional information or the loss function.

Response: Since the baseline using the cross-entropy loss had access to these new labels as well, the improvement is necessarily from the loss function.

Questions:

1. According to my understanding, applying the surrogate loss requires labeling the output from the given model. Then the model is further trained by the learned surrogate loss. Thus some extra information is introduced. Can the Cross entropy loss and Maxprob have the same information?

Response: The cross entropy baseline did have the same information - we finetuned a model using the cross-entropy loss on the annotations. However, it would not be possible for Maxprob to also benefit from this information because the annotations form a binary classification task, while the predictor (from which Maxprob is derived) is trained to generate decontexualizations.

2. What is the format of the rejector?

Response: The format for the rejector is described in section 7.2 (surrogate loss), and it is trained on the decontextualization rejection task described in section 7.1. At a high level, its input is a sentence and a candidate decontextualization, and its output is a real number whose sign corresponds to acceptance or rejection.

3. Why is the F1 score not applied for clear comparison?

Response: The F1 score relates to precision versus recall, while instead we are interested in precision versus coverage. We believe that it is very natural and easy to compare the methods by plotting precision versus coverage (which we do in Figures 4, 5, and 6).

4. For the std comparison it is not very clear whether rejection loss is much better than Maxprob in Figure 4 and in Figure 5.

Response: We respectfully disagree with the reviewer. The curve for the rejection loss lies above that of Maxprob in both figures, and in Figure 4, it lies significantly above that of Maxprob.

Thank you for your encouraging review. We will take your suggestions into account when preparing the final version. Below please find responses to specific questions.

1. The method is only applied to the decontextualization problem when it actually seems to have much more broad applicability. I can think of several LLM applications where the ability to increase precision by rejecting would be useful. For example, we could try to learn a rejector that cuts down on hallucinations in summarization or text simplification. More exploration of this technique beyond the decontextualization problem would make it more impactful.

Response: As discussed in the conclusion, annotation for the LLM setting is expensive and this limits our empirical evaluation. We therefore provided additional experimental evaluation on image classification tasks, where checking the correctness of the predictor is easy. We leave further experimental evaluation for future work, but observe very strong results on our current LLM experiments.

2. When $\mathcal{R} = R_{\mathrm{all}}$, can't we use a cost-sensitive surrogate loss plus usual (Bayes) consistency results? I.e., I don't understand the following claim in Appendix C given that the paper's results only consider $\mathcal{R} = \mathcal{R}_{\mathrm{all}}$.

In general, I think the paper could use more presentation on why the naive approach of directly training the rejector to predict the labels $a$ using a standard cost-sensitive surrogate loss doesn't work, since the results only consider the $\mathcal{R} = \mathcal{R}_{\mathrm{all}}$ case. That, or some results for linear or 1-layer-NN rejectors, as in Awasthi et al. (2022), would strengthen the theoretical part of the paper.

Question 1. Why do we care about $\mathcal{H}$-consistency even though we only consider the space $\mathcal{R}_{\mathrm{all}}$ of all measurable functions? It would be helpful to further emphasize this earlier in the paper as a major difference / novelty / reason why more naive baselines don't work (see earlier comment under weaknesses).

Response: Our surrogate losses benefit from $\mathcal{R}$-consistency bounds with the hypothesis set $\mathcal{R}$ extending beyond the family of all measurable functions $\mathcal{R}_{\mathrm{all}}$. This includes, for example, the family of linear models and one-hidden-layer neural networks considered by Awasthi et al. (2022). For simplicity, we presented results only for the family of all measurable functions. However, our proofs and results can be directly extended to other hypothesis sets. In the final version, we will include these $\mathcal{R}$-consistency bounds.

While the cost-sensitive surrogate loss benefits from Bayes-consistency, it does not benefit from $\mathcal{R}$-consistency bounds. As pointed out by Awasthi et al. (2022), Bayes-consistency does not supply any information about learning with a typically restricted hypothesis set, which of course would not contain all measurable functions. Furthermore, it provides no guarantee for approximate minimizers since the rate of that asymptotic convergence could be arbitrarily slow.

3. No ablation on $\alpha$ even though it's important in the bound (Thm 5), and different experiments use different values (e.g., 4 in main text experiments, 3.5 in appendix vision experiments).

Response: In our experiments on image classification, the value of $\alpha$ is selected using cross-validation, a method typically applicable in practice. In our future work, we aim to explore a more principled approach for choosing $\alpha$. A promising direction involves establishing data-dependent bounds, which would act as a useful guide for selecting $\alpha$ across different datasets.

4. Small typo in display at bottom of pg4; $\leq 0$ should be in the subscript of $\mathbb{I}$.

Response: Thank you, we will fix this in the final version.

5. Small typo in display at bottom of pg4; not enough parentheses.

Response: Thank you, we will fix this in the final version.

6. Minor semantics/notation: $r(x, f(x))$ reads like a score for rejecting, so the notation reads like $r > 0$ $\implies$ reject, when it's actually being used in the opposite sense. This tripped me up several times. I know this notation is inherited from Cortes et al. but I think it's a bit confusing.

Response: We are indeed staying consistent with the notation from the literature.

We have uploaded a revised version with a newly added Appendix D. The main results (on p.18), Theorem 7 and Corollary 8, demonstrate that our surrogate losses benefit from $\mathcal{R}$-consistency bounds with any bounded hypothesis set, including the family of linear models and one-hidden-layer ReLU networks considered in (Awasthi et al., 2022) as special cases. In the initial version, we mentioned before Section 5.5 "Similar results can be derived for other families of functions $\mathcal{R}$, such as that of linear functions or neural networks with one hidden-layer as in (Awasthi et al., 2022)", but did not include them for simplicity. We believe these newly added results will further strengthen the theoretical part of the paper and address the reviewer's concerns.

Note that there is a lack of similar guarantees for cost-sensitive surrogate losses with respect to the induced rejection loss. Furthermore, our surrogate loss outperforms the cost-sensitive baseline in our image classification experiments.

Note that one term in the sum has $\mathbb{I}[r > 0]$ and the other term has $\mathbb{I}[r \le 0]$, so only one term can be nonzero. This yields the first inequality.

Thank you for your appreciation of our contribution. We have carefully addressed all the questions raised, especially regarding the consistency of our mathematical results. Please find our responses below, and we respectfully request the reviewer to reconsider the overall rating.

Weaknesses:

1. Although GitHub repo links and other identifying information cannot be written in a paper under review, I did not see any indication of the intention to make the code public, nor is it provided in supplementary materials.

Response: We are hoping to release code for the final version and are waiting for approval.

2. Page 4. First equation/inequality.

Response: As pointed out by reviewer KRjg, the first inequality indeed holds. Thus, we are still giving an upper bound. We will replace the second inequality with equality (we were simply staying consistent with Cortes et al.). The last relation in the derivation holds because a max of two positive real numbers is upper-bounded by their sum.

3. There is no test set. Only train and validation, where cross-validation is used so training algorithm sees all data.

Response: We perform k-fold cross validation, and thus the training algorithm does not see the data it is tested on. Additionally, in the decontextualization experiment with LLMs, we do not tune any hyper-parameters for our proposed method.

Questions:

1. page 5. it is said that "underlying scores are not favourable for that precision level". Why is that?

Response: We would like to point out that we have instead written “not necessarily favorable.” This is because the same scores are used for each precision level; the learning algorithm was not trained to target a specific precision level. On the other hand, in our surrogate loss, we are targeting a specific precision level through the value of $c$.

2. These are possibly standard deviation bars in figures 4 and 5. How were they generated? Is it from different folds of cross-validation?

Response: As described in their respective captions, the standard deviations in Figure 4 are from the different cross-validation splits, and the standard deviations in Figure 5 are from different training runs.

3. Are Maxprob and cross-entropy trained on different models? Why is that?

Response: These are very slightly different - Maxprob was derived from an existing decontextualization model, and we chose to use T5X for all of our fine-tuning as it was the closest model available to us.

Minor Typing / Formatting / Clarity issues:

1. page 4. last equation "<=0" should be in the subscript

2. please recheck format for citations: some citations use et al while other list all authors.

Response: Thank you, we will fix these in the final version.