LitCab: Lightweight Language Model Calibration over Short- and Long-form Responses

We truly value your constructive feedback and we've carefully addressed each of your concerns, as outlined below:

1. The paper lacks discussion and comparison of some related works.

ITI [1] focuses on enhancing the truthfulness of LLMs and their main purpose is not strengthening the calibration performance of LLMs, so we didn’t take it as our comparison. Due to insufficient time, we only compared LitCab with ITI on TruthfulQA, which was implemented in their code. The results are shown in the table below.

We can see that LitCab outperforms ITI in calibration across all metrics.

Tuned Lens [2] offers an analytical approach to comprehend the inference process of transformers, potentially aiding in refining models. In contrast, LitCab tackles a challenge in language models - calibration - and provides an effective, efficient solution. Tuned Lens has the potential to improve the calibration performance of language models, e.g. estimating the model confidence from the signals of Tuned Lens, and we will consider this aspect in our future work. We will carefully discuss all the work and their relation to our method in the revision.

2. The paper lacks results compared with efficient fine-tuning methods.

In Sections 6.3 and 6.4 (highlighted in green), we introduced and compared LitCab with Label Smoothing, where the LM is fine-tuned using LoRA. Results in Table 1 indicate that it does not yield satisfactory calibration performance, possibly due to the small size of the training data leading to model overfitting. In contrast, LitCab can leverage small training sets for improved calibration, suggesting that LitCab achieves superior data efficiency compared to label smoothing.

3. Using GPT-4 to generate training data

To clarify, we only use GPT-4 to evaluate the correctness of the LM's responses. All the incorrect responses for training LitCab are generated by the LM itself.

4. What are the main advantages of the proposed method? How does it improve LLMs qualitatively or quantitatively?

In response to the observation of the baseline's relatively low ECE, which falls within the range of approximately 0.1 to 0.3, it is important to note that the typical ECE range is not confined to 0 to 1, as randomness in model confidence and correctness alone can yield an ECE of 0.25. However, our proposed method, LitCab, has demonstrated a significant and noteworthy impact on calibration. Table 1, highlighted in green, serves as compelling evidence, showcasing LitCab's effectiveness in achieving a remarkable 25% decrease in the average expected calibration error—from 0.118 to 0.089. This substantial reduction underscores the significant improvement LitCab brings to calibration, emphasizing its practical importance and contribution to the field.

5. Why do you use different thresholds (e.g., cov@50, cov@90, cov@60) for different datasets?

This is because language models demonstrate varied performance across different datasets. Cov@p, as defined, assesses the model's recall by identifying the largest percentage denoted as 'c,' for which the most confident c percent of predictions exhibit accuracy surpassing the threshold 'p.' It is crucial to note that a lower or higher value of 'p' can result in meaningless comparisons; for example, all models will achieve 100% when p is set to 0 and 0% when p is set to 1.0.

To illustrate, consider SciQ, which is relatively easy for LMs. If we set a lower value of 'p' in cov@p (e.g., 50), it results in all 100% for all models in cov@50. This poses a challenge in utilizing this metric for comparisons. Therefore, in alignment with the approach outlined by Tian et al. (2023) [3], we opt for a relatively high value of 'p' to facilitate more meaningful and informative comparisons.

It is important to emphasize that we do not solely rely on this metric (due to its aforementioned limitations), we also report acc@q, ECE, and Brier score, to include a comprehensive set of metrics for evaluating the calibration performance of models.

Reference

[1] Inference-Time Intervention: Eliciting Truthful Answers from a Language Model. arXiv 2023.

[2] Eliciting Latent Predictions from Transformers with the Tuned Lens. arXiv 2023.

[3] Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D Manning. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. Arxiv 2023.

We hope our explanations and modifications have addressed your concerns. If you find them satisfactory, we kindly ask you to take these improvements into consideration when reassessing our work. We are open and would greatly appreciate any additional comments, follow-up questions, or suggestions related to our work.

2. There is only 1 dataset with long-form responses. More datasets, such as ELI5, WebGPT, ASQA, and QAMPARI should be considered.

Thank you for your suggestion. We followed your advice and conducted the experiment on QAMPAR because there are several issues when experimenting with other datasets. Regarding ELI5, collecting support documents for correctness evaluation is extremely time-consuming, and unfortunately, we won't be able to do that due to the limited rebuttal time. However, we will definitely consider experimenting with ELI5 in the future. As for WebGPT, there is currently no available and consistent source for checking the correctness of model responses, given the open-ended nature of the questions. While quoted sentences are provided, they are not sufficient for verifying the knowledge in various model responses. ASQA primarily inspects the model's capability on ambiguous questions, which includes the uncertainty of questions, and it's not within our scope. This leaves QAMPARI, where the correctness of model responses can be verified by comparing them with provided references. LitCab demonstrates its superiority in this task, based on the results below

Results of LitCab and baselines on CaT

Furthermore, to evaluate LitCan on more tasks, we introduce a new task for long-form generation—WikiGen, where the LM is tasked to generate Wiki-style descriptions for entities, with an average generation length of 104, sourced from the fact verification dataset FEVER [1]. Compared with BioGen which is limited to biographies, WikiGen covers a wider spectrum of subjects beyond individuals, including events, concepts, and objects, resulting in more diverse topics. LitCab again obtains the best calibration performance, as evidenced by the results presented in the following table:

Results of LitCab and baselines on CaT

We also evaluate seven LLMs using both QAMPARI and WikiGen:

Different calibration metrics of popular LMs computed over CaT

Reference

[1] James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a Large-scale Dataset for Fact Extraction and VERification. NAACL 2018.

We sincerely appreciate your constructive feedback, and we have addressed each of your concerns as outlined below:

1. Coherence issues.

In this work, we present four primary contributions. Firstly, we introduce LitCab, a lightweight calibration mechanism. Its effectiveness is demonstrated through comparisons with various competitive approaches. Secondly, we introduce CaT, designed to assess text generation tasks with responses of varying lengths. Thirdly, we extensively evaluate the calibration of seven state-of-the-art open-source language models on CaT, deriving three key findings related to model size, model family, and model fine-tuning. Fourthly, we formulate an evaluation strategy for paragraph-level model generations. In the revised introduction, we have highlighted the significance of the third contribution, found in the second paragraph from the bottom, which is marked in blue.

3. The proposed baselines could not be evaluated in all cases.

To provide comparisons, we have to create training samples for Label Smoothing and Temporal Scaling in paragraph-level tasks where ground truth is unavailable. This involves gathering accurately generated claims, assessed using GPT-4.

We present the results of Label Smoothing and Temporal Scaling on paragraph-level tasks in Table 1 in our revised version (highlighted in blue). The results suggest that both methods adversely affect the LM's calibration performance. This is attributed to tuning the temperature or fine-tuning the entire LLM based on generated samples leading to overconfidence.

4. How would the use of Litcab fare on arithmetic and commonsense reasoning benchmarks?

In this paper, we focus on evaluating the calibration of LMs on outputs of varied lengths. Therefore, we design LitCab in the context of generation tasks and conduct experiments related to such tasks.

Following your suggestion, we further conduct experiments on the CommonsenseQA task under the multiple-choice setting. We compare the results of the original LM, self-consistency and LitCab. LitCab demonstrates superior calibration performance compared to both the original language model and self-consistency, as evidenced by its lower ECE and Brier score, as outlined below:

5. There is no evidence shown in the paper that supports "advantage of requiring a reduced number of data samples for training, leading to enhanced computational and data efficiency".

We apologize for our imprecise claim. We have rephrased the corresponding description in the revision: LitCab achieves superior computational efficiency with fewer tunable parameters, compared with the fine-tuning-based baselines.

Dear Reviewer Y9cb

Thank you for your dedicated review efforts. As the discussion stage deadline approaches, please feel free to raise any concerns or seek clarification on our manuscript. Any feedback is greatly appreciated and will help us further enhance this work.

Thank you,

Authors

We hope that these clarifications address your concerns. We believe the paper has been significantly improved thanks to your comments.

If any points remain unclear, or if you have additional questions or suggestions, please feel free to share them.

In light of these improvements and clarifications, we kindly request that you consider adjusting your scores.

Thank you once again for your time and effort in reviewing our work. Your valuable insights are greatly appreciated.

Thank you for your thoughtful comments and suggestions. We have carefully considered all of them and addressed each one in the revised version of our paper. We truly believe that these modifications and clarifications significantly improved our work.

1. It’s not clear how to select the incorrect answers.

For phrase- and sentence-level tasks where references are available, we use the references directly as positive samples. We then repeatedly sample model generations, treating incorrect ones, evaluated by GPT4, as negative samples. For the paragraph-level tasks, we directly instruct each LM to generate biographies or descriptions for the given people’s names or entities. Subsequently, we categorize the correct claims as positive samples and the incorrect ones as negative samples. We have included this description in the final paragraph of section 4 in our revised version–highlighted in blue.

2. It’s not clear why adding a linear layer on the last-layer hidden states is the best option.

This is because we aim to prioritize computational efficiency in the design of the lightweight calibration mechanism. Fine-tuning the LM through LoRA involves gradient back-propagation through all layers of LM. In contrast, training LitCab only necessitates running the LM's inference process to acquire the last-layer hidden states and logits. The single linear layer can then be trained based on these gathered features.

In our pilot study, we experimented with both the second-to-last layer and the last layer's hidden states and found that using the last layer performed better. This is likely since because the last layer is closest to the logits and encodes more information about the future token. Exploring the optimal layer or combinations of the layers is a topic we intend to delve into in our future work.

In Section 6.3 and 6.4 (highlighted in green), we compare LitCab with Label Smoothing fine-tuning with LoRA. Results in Table 1 indicate that Label Smoothing does yield satisfactory calibration performance, possibly due to the small size of the training data leading to model overfitting. In contrast, LitCab can leverage small training sets for improved calibration, suggesting that LitCab achieves superior data efficiency compared to label smoothing.

3. How to handle the different importance of tokens?

We answer your comment in two points. Firstly, this aggregation method has been studied and shown as effective in approximating the sentence-level probability by Kuhn et al., 2023 [1], and Si et al., 2023 [2], we aggregate token probabilities to calculate the sentence probability in our work.

Secondly, we argue that there is no straightforward definition of what constitutes an important token, as the token importance heavily relies on the context, such as the question, and it could easily change if any tokens related to the question are altered. In reference to your example, the model may generate an answer like "The 2022 Oscar best actress is Michelle Yeoh." If we only focus on the individual’s name, the incorrect generation of "2022" might be easily overlooked.

In light of these considerations, we chose to treat all tokens equally for the sake of simplicity and generalizability to text of varied lengths in this work. We are also open to and appreciate any suggestions from this perspective.

Reference

[1] Kuhn, Lorenz, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. ICLR 2023.

[2] Si, Chenglei, Zhe Gan, Zhengyuan Yang, Shuohang Wang, Jianfeng Wang, Jordan Boyd-Graber, and Lijuan Wang. Prompting gpt-3 to be reliable. ICLR 2023.

Dear Reviewer 7atx

Thank you for your dedicated review efforts. As the discussion stage deadline approaches, please feel free to raise any concerns or seek clarification on our manuscript. Any feedback is greatly appreciated and will help us further enhance this work.

Thank you,

Authors