DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models

We thank Reviewer Cc1D for the constructive comments!

subtracting early layer logits make sense, but this might also be related to residual connection from early layers to final layer. Have you tried just removing the residual connection between the premature layer and final layer? Would that be helpful without too much decrease in inference speed?

Removing the residual connections of the premature layer could potentially be another method to downplay the early layers and emphasize the higher layers, without causing any latency. We quickly tried some initial experiments with this idea. However, we found that once we remove the residual connection from one of the attention layers or MLP layers, the decoding output will become a sequence of repeated tokens. For example, when removing the MLP residual connection of layer 12, we got:

Question: Eliza's rate per hour for the first 40 hours she works each week is $10. She also receives an overtime pay of 1.2 times her regular hourly rate. If Eliza worked for 45 hours this week, how much are her earnings for this week?

Model Output: s that that that that s that s s s that that s that that s that that s s s that that that s that that that that s that that that that that that …

We suspect that the removal of residual connections causes a significant distribution shift for the hidden states, thus resulting in out-of-distribution inputs for the next layer and subsequently unusual model behavior. It seems more difficult to manipulate the continuous hidden states, compared to manipulating only the output logits as we did in DoLa, as the output logits over discrete vocabularies are more explainable. However, we believe that this could be a direction that is worth exploring more carefully in the future. Additional fine-tuning may be required to solve the distribution shift issue.

what's the inference speed of your method compared to the baseline?

For the latency analysis, we use the 817 examples from TruthfulQA with the default 6-shot in-context demonstration prompt which has an average input length is 250.3 after concatenating the prompt with the questions. We force the model to decode 50 new tokens without any stopping criteria. All the experiments were run on the machine with NVIDIA 32G V100s equipped with 40-core CPUs of Intel(R) Xeon(R) Platinum 8168 CPU @ 2.70GHZ. We run the models with 16-bit floating point and batch size = 1. For 7b/13b/33b/65b models, we use 1/2/4/8 V100 GPUs, respectively. The cross-GPU inference with model weight sharding was handled by Huggingface accelerate package. [1]

Here is the latency (ms/token) and throughput (tokens/sec) for 4 models.

Latency:

Throughput:

We can see that the speed is slightly slower than the vanilla decoding baseline by 1%~8%, which is in an acceptable range. This result suggests that DoLa is a practical decoding strategy that has the potential to be applied to real-world applications.

[1] https://huggingface.co/docs/accelerate/concept_guides/big_model_inference

Thank you once again for your insightful review! Since the discussion period will end in less than two days, we wanted to make sure that we have adequately addressed the issues you raised. We also hope that our additional experiments have addressed most of your concerns. We would appreciate your feedback on our responses!

Decoding from every layer seems expensive. I’m surprised by the efficiency results presented in Section 4.4. Can the authors provide more details on, e.g., how is this measured and on what tasks/hardware, and the generation lengths?

In the latency analysis in Section 4.4, we use the 817 examples from TruthfulQA with the default 6-shot in-context demonstration prompt which has an average input length is 250.3 after concatenating the prompt with the questions. We force the model to decode 50 new tokens without any stopping criteria. One of the reasons we make DoLa so efficient may be the fact that we use batched tensor operations for computing JS-divergence all at once for all the candidate layers together.

All the experiments were run on the machine with NVIDIA 32G V100s equipped with 40-core CPUs of Intel(R) Xeon(R) Platinum 8168 CPU @ 2.70GHZ. We run the models with 16-bit floating point and batch size = 1. For 7b/13b/33b/65b models, we use 1/2/4/8 V100 GPUs, respectively. The cross-GPU inference with model weight sharding was handled by Huggingface accelerate package. [3]

[Important] Following the above, latency is only one aspect of efficiency and a determining factor of whether DoLA “can be widely applied with negligible cost.” It would be great if the authors can quantify DoLA’s impact on throughput and memory overhead too.

Thanks for the suggestion. We conduct additional experiments to measure throughput and memory overhead.

Memory Overhead:

To measure the overhead, we calculate (a) the occupied GPU memory before the first forward pass (b) the peak GPU memory during the forward passes. And then we can compute the memory overhead by (b) - (a), or the proportion of overhead [(b) - (a)] / (a) in %. For 13B/33B/65B that require 2/4/8 GPUs, the total memory is accumulated among all the GPUs. The results are shown below:

We can see that during the forward pass of LLaMA-7B, the overhead for vanilla decoding is 2.5% while DoLa requires 3.6%. There is only 1.1% difference for the memory overhead between Vanilla and DoLa. For 13b/30b/65b models, the difference is even smaller than 1%. This result shows that the difference in memory overhead between DoLa and the vanilla decoding baseline is still negligible.

Throughput:

Here is the throughput (tokens/sec) for 4 models.

We can observe that the differences are generally less than 7%, similar to the extent of increased latency we have shown in Section 4.4.

[3] https://huggingface.co/docs/accelerate/concept_guides/big_model_inference

We thank Reviewer VLmJ for the constructive comments!

[Important] In the example in Figure 2, almost all layers selected by DoLA will be the wording embedding layer.

In Figure 2, when predicting important tokens that are related to factual information, the higher part of the layers are more likely to be selected. When predicting the easy-to-predict tokens like function words, the word embedding layer is more likely to be selected. The claim that “almost all layers selected will be the wording embedding layer” only applies to easy-to-predict tokens, not all of the tokens.

In such cases, assuming $q_0(x_t)$ has a substantial amount of mass on $x_t$, DoLA is essentially discouraging the model from generating the same token as the previous one. Maybe I’m missing something obvious, but I am not entirely convinced by the “factuality happens at higher layers” narrative.

Because LLaMA’s word embedding layer is NOT tied with the LM head [1], when feeding the word embedding of the previous token directly into the LM head, the output distribution ($q_0(x_t)$) would not put a substantial amount of mass on the previous token ($x_t$) at all. Instead, the output distribution will be closer to a non-contextual prediction of the next token ($x_{t+1}$) only conditioned on the previous token ($x_t$), similar to the idea of a bi-gram language model ($P(x_t+1|(x_t))$). Thus, DoLa is NOT discouraging the model from generating the same token as the previous one. I believe this is a misunderstanding.

[Important] Following the above, have the authors compared to a baseline that always selects the 0-th layer? I won’t be surprised if it achieves strong performance, since it is doing exactly the same thing as DoLA on most generation steps.

Contrasting with the 0-th layer indeed improves the performance. Following the above discussion, the effects of the 0-th layer prediction will be similar to a bi-gram LM. Contrasting the final predictions with the bi-gram LM prediction should be beneficial, as the bi-gram predictions contain the non-contextual language statistics bias learned from the pretraining data. Contrasting the final layer with the 0-th layer should be able to remove this non-contextual bias from the final contextualized predictions, emphasize the contextual information that is later injected in the middle layers, and thus make the final prediction unbiased and better.

In the above table, we show the result of DoLa with only contrast to the 0-th layer. In general, using only the 0-th layer already improves the performance compared to the vanilla decoding baseline. DoLa incorporates information from multiple layers can further increase the scores.

[1] https://github.com/facebookresearch/llama/issues/138; We also have verified this by manually comparing the embedding weights and the LM head weights of LLaMA. They are completely different weights.

Weakness: I don’t see a way to apply DoLA to proprietary models.

DoLa is generally applicable to transformer-based LLMs. The accessibility of blackbox proprietary models like ChatGPT and GPT3/4 is out of the scope discussed in our paper.

We believe that under the scope of academic research, we should encourage research that makes progress in improving or understanding the fundamental mechanism of these large models, instead of considering whether the research favors closed-source proprietary models.

In fact, if all the researchers were encouraged to focus only on methods (e.g. prompting) that work for black box API LLMs, then we would not have a chance to discover the underlying mechanism inside the black box of LLMs, and the possible improvement that can be made would be very limited as we are restricted to surface-level interactions.

Weakness: It would be great to investigate DoLA’s impact on the language model’s generation quality

Thanks for the insightful suggestion! We conduct an additional study of the quality of generated text using GPT4, given the fact that several prior studies [4][5] have shown the great potential of GPT-4 to serve as an alternative to human evaluation. And the effect is stable over different prompts and instructions. [6]

We adopt the pairwise evaluation code from Vicuna QA. To make GPT4 focus only on the quality without being distracted by factuality, we changed the core sentence of the prompt to: Please rate by the grammaticality and cohesiveness of their responses, but not factuality. You are not required to verify the factual accuracy of the answers. Each assistant receives an overall score on a scale of 1 to 10, where a higher score indicates better quality.

By using the prompt above, we observed GPT-4 can rate the answers based on grammaticality and cohesiveness without checking the factual correctness. The results are shown below, where the scores are the average scores from 80 questions in Vicuna QA, on a scale of 1 to 10.

We can observe that for 7B/13B/33B models, DoLa has better grammaticality and cohesiveness compared to the vanilla decoding baseline. For the largest 65B model, DoLa achieves a score that is almost the same as vanilla decoding. We conclude that when evaluating text generation quality without considering factuality, DoLa is still on par with (65B) or better than (7B/13B/33B) vanilla decoding.

[4] Chiang, Cheng-Han, and Hung-yi Lee. "Can Large Language Models Be an Alternative to Human Evaluations?." arXiv preprint arXiv:2305.01937 (2023).

[5] Liu, Yang, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. "G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment." arXiv preprint arXiv:2303.16634 (2023).

[6] Chiang, Cheng-Han, and Hung-yi Lee. "A Closer Look into Automatic Evaluation Using Large Language Models." arXiv preprint arXiv:2310.05657 (2023).

Figure 2 is nice; I wonder whether the authors have any quantitative results on this.

Thanks for the inspiring suggestion! We include an additional quantitative study, where we use the validation set of the CoNLL-2003 name entity recognition dataset [2] with 3.25K examples. We calculate which layer has the largest JS-divergence with the final layer when LLaMA-7B predicts the next token with teacher forcing (we simply call this layer the "critical layer" for short). We subdivide the results into two parts by whether LLaMA is predicting an entity token or a non-entity token.

We can find that 75% of the time the critical layer will be layer 0 when predicting non-entity tokens. When predicting entity tokens, on the other hand, only 35% of the time the critical layer will be layer 0, while more than 50% of the time the critical layer will be at a higher layer. This experiment can quantitatively support our observations in Figure 2.

Note that we use teacher forcing to send the ground truth into LLaMA to predict the next word for each token in the sentence. And the ground truth sentences are not generated by LLaMA. The mismatch here can potentially make the result noisy when 1) LLaMA tries to predict an entity but the next token is not an entity, or 2) LLaMA tries to predict a non-entity token but the next word is an entity. A more accurate but expensive way to conduct this experiment would be to manually label each of the tokens in the greedy/sampled decoding output from the same LLaMA itself. However, from the current experiments we have already seen such a trend in the NER dataset.

[2] Sang, Erik Tjong Kim, and Fien De Meulder. "Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition." In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pp. 142-147. 2003. https://huggingface.co/datasets/conll2003

Thank you once again for your insightful review! Since the discussion period will end in less than two days, we wanted to make sure that we have adequately addressed the issues you raised. We also hope that our additional experiments have addressed most of your concerns. We would appreciate your feedback on our responses!

In the experimental setup, a bucket is selected first, and then the premature layer is selected later. This approach can save time. However, have you tried the dynamic layer selection with brute force? Is this approach able to achieve better results?

Here we can discuss two kinds of brute force:

1. Static layer selection with brute force, namely DoLa-static: Using the same premature layer for the entire decoding process. The layer was selected by the overall scores on a validation set. The results are shown in Table 10, and 11. We can see dynamic DoLa is slightly better than DoLa-static on CoT reasoning tasks, while DoLa-static can be slightly better than dynamic DoLa on FACTOR.

2. Dynamic layer selection with brute force: It is a very complex search problem if we want to apply the brute force idea in every time step of dynamic DoLa. Given a validation example (e.g. question in GSM8K), we need to finish decoding the full-sentence answer (e.g. CoT reasoning path in GSM8K) before calculating the score (e.g. exact match scores in GSM8K). For example, if we have 20 candidate layers to select, and we want to generate 50 new tokens, the possible combinations of dynamic layer selection strategy will become $20^{50}$, which is not feasible to apply brute force. Also, we cannot obtain the instant reward for each of the decoding steps until we generate the full sentence, so it is hard to do something like a beam search to eliminate the low-score search paths in the dynamic layer selection process. Even if we somehow successfully obtain the optimal path for a specific validation example, it would be hard to transfer it to unseen examples in the testing set. We believe that this is still an open question to apply brute force to dynamic layer selection.

For some datasets, e.g, FACTOR-News, the 0-th layer is a good premature layer, as shown in Figure 6. However, for TruthfulQA, the selected layer is closer to the final layer. What do you think the reason is?

The examples in TruthfulQA are mainly short answers with difficult crucial facts. For example:

• Q: Where did fortune cookies originate?
• Correct Answer: Fortune cookies originated in San Francisco.
• Incorrect Answer: Fortune cookies originated in China.

We reasonably surmise that the crucial facts in TruthfulQA require focusing more on directly contrasting with the layers right before the topmost layers where the factual knowledge is located.

In contrast, examples in FACTOR are long paragraph prefixes with 4 possible long sentence completions as the multiple choices, which contain many “non-fact” tokens that require other general knowledge (e.g. linguistic knowledge) that are not necessarily located in the topmost layers. In this case, contrasting with the lower part of the layers can better take care of all the tokens in the whole sentence. Although this is just our hypothesis, the best bucket of candidate layers of FACTOR/GSM8K/StrategyQA are all the same: the lower part of the layers, which also transfer well to VicunaQA. All of these datasets have long-sentence answers.

After reading the paper and the following reference paper, it is still a little harder for me to conclude that this approach for contrastive decoding is better than the previous one proposed by Li et al. (2022).

Based on the above discussion in (1/3), the reference paper [1] is already different from the original setting of Li et al. (2022) [2] by introducing the hyperparam $\beta$ to improve its performance. It should NOT be a valid reason to claim that DoLa is not better than Li et al. (2022) [2] simply based on the numbers in the reference paper.

The contribution of $\beta$ proposed in [1] should be orthogonal to the topic discussed in our DoLa paper. We believe that carefully tuning $\beta$ can also improve the GSM8K scores in our paper both for the CD baseline and the proposed DoLa. Since the reference paper [1] does not have open-source code yet, we will leave adding the $\beta$ hyperparameter into our method in the next version.

Please note that actually this reference paper [1] was uploaded to arXiv one week before the ICLR submission deadline, so we do not have the chance to incorporate this proposed method in [1] to improve either our baseline or our proposed DoLa. We also note that the ICLR guidelines for considering prior work only apply to papers arXiv-ed 30 days before the ICLR deadline. [3]

The results are a little inconsistent. In Table 1, for TruthfulQA (MC1) with LLaMa-33B, we can see that the proposed method (DoLa) have worse performance than the other two baselines (basic decoding and contrastive decoding). It is better to discuss a little about it.

In TruthfulQA, the MC1 metric is relatively sensitive compared to MC2/MC3, because MC1 is a "winner takes all" metric. If any one of the false answers has a higher score than the best true answer, then it will return a 0.0 score, otherwise 1.0. Thus, small fluctuations of the model outputs only on a single example can still change the score from 1 to 0, or from 0 to 1. On the contrary, MC2 considers whether or not the normalized probability mass for all correct answers is higher than that of the false answers. MC3 considers whether or not each correct answer has higher scores than all false answers, and then averages the results over all correct answers. Thus, MC2/MC3 will be more stable to the noises as they consider all the correct answers together. [4]

It is reasonable to get inconsistent results especially when the metric itself is sensitive to noise. And this is the reason why we have tried a lot of experiments in our paper to show the overall trend of improvements across different datasets/tasks.

[1] O'Brien, Sean, and Mike Lewis. "Contrastive decoding improves reasoning in large language models." arXiv preprint arXiv:2309.09117 (2023).

[2] Li, Xiang Lisa, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. "Contrastive decoding: Open-ended text generation as optimization." arXiv preprint arXiv:2210.15097 (2022).

[3] https://iclr.cc/Conferences/2019/Reviewer_Guidelines

[4] The official implementation of MC1/MC2/MC3 from TruthfulQA: https://github.com/sylinrl/TruthfulQA/blob/main/truthfulqa/models.py#L540

We thank Reviewer NRik for the constructive comments!

The baseline for contrastive decoding is not well explored enough. In some papers, for example, "CONTRASTIVE DECODING IMPROVES REASONING IN LARGE LANGUAGE MODELS", A smaller model (1.5B parameters) is as the amateur model. The better results are shown in that paper. For example, contrastive decoding performance on GSM8K with Llama-65B is 56.8. It is even better than the proposed method, DoLa (54.0).

The Contrastive Decoding (CD) baseline in the reference paper [1] has two significant differences compared to our CD baseline.

1. They train a small LLaMA-1.5B model as the amateur LM: This paper [1] pretrains a Llama-1.5B model that is not publicly released . As we do not have that 1.5B model, we use the 7B model in our experiment. After the conference submission deadline, Sheared-LLaMA was released. Therefore we try to use the 1.3B and 2.7B Sheared-LLaMA and 3B/7B OpenLLaMA as the amateur LMs. These are all of the publicly released small LLaMAs that share the same vocabulary as LLaMA (so that we can contrast them on the same vocab set). The results on GSM8K are shown below.

We can see that using a small amateur LM, especially the 1.3B one, can improve the scores for contrastive decoding compared to using the 7B one as the amateur LM. However, the scores are still not better than DoLa. We suspect that the choice of the amateur LM is critical. The paper [1] may have put some effort into finding a suitable LLaMA-1.5B model that hits the sweet spot. Our experiments on these open-sourced small LLaMA models still cannot match the performance in [1], showing the difficulty in choosing amateur LMs for CD.

2. They introduce an extra hyperparameter $\beta$ that does not exist in the original CD paper: While we only follow the original version of contrastive decoding [2], in Section 2.1 of this paper [1], the authors propose to add an extra hyperparameter $\beta$, the strength of the amateur penalty, for better balancing between expert logits and amateur logits. Thus, the contrastive decoding formula will not be the same as the one in the original CD paper, unless $\beta=\infty$.

The reviewer argues that the 65B CD result of GSM8K is 56.8 in [1] which is much better than our 65B CD baseline (44.0) and our 65B DoLa (54.0). However, we argue that 44.0 is already a reasonable number that consistent with the results in [1] by the following evidence. In Table 1 from [1], the high GSM8K score of 65B (56.8) is under the setting of $\beta = 0.5$, while it drops significantly to 44.6 when setting $\beta = 1.0$, which is very close to our 65B CD baseline (44.0) which essentially equals to the situation of setting $\beta=\infty$.

By the above evidence, we argue that the high score in [1] is probably achieved by carefully tuning $\beta$ at a sweet spot. If following the original version of CD [2], our score (44.0) already matches the score shown in [1] with $\beta=1$.

[1] O'Brien, Sean, and Mike Lewis. "Contrastive decoding improves reasoning in large language models." arXiv preprint arXiv:2309.09117 (2023).

[2] Li, Xiang Lisa, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. "Contrastive decoding: Open-ended text generation as optimization." arXiv preprint arXiv:2210.15097 (2022).

Thank you once again for your insightful review! Since the discussion period will end in less than two days, we wanted to make sure that we have adequately addressed the issues you raised. We also hope that our additional experiments have addressed most of your concerns. We would appreciate your feedback on our responses!

Dear Reviewer,

we would like to thank you again for your valuable efforts, time, and suggestions! We believe we have answered your questions in the above responses. Considering that the deadline for reviewer-author discussions is approaching, please let us know if you have any further concerns or questions about our work. We will be happy to answer any questions you may have!

Dear Reviewer NRik,

The author-reviewer discussion period will end in less than 12 hours. Please let us know if you have any concerns or questions. We would be happy to address them!

“DoLa simplifies hyperparameter search space: it needs only 2-4 bucket tests, almost 10x fewer than the 16-40 tests needed in DoLa-static” - This savings at hyperparameter search time is surely more than negated by the ~10x JS-divergences that need to be computed for every inference to select the optimal layer for non-static DoLa, no?

We agree that DoLa-static can slightly reduce the inference latency of computing JSD in DoLa, at the cost of more validation runs. The advantages of DoLa and DoLa-static are depending on the use case.

• If it is a popular downstream application with millions of users and a fixed downstream task with a reliable validation set available, then the optimal inference time is the priority consideration and it is worthy to have tens of times more validation runs. In this case, DoLa-static is a good choice.

• If it is a customized application that has a moderate amount of users (e.g. users in a specific small region, which is the usual case for even popular apps), then the cost of adapting the model to multiple user groups is significant, and an inference speed of 1.08x slower is relatively neglectable. In this case, DoLa will be preferred over DoLa-static.

From the perspective of efficiency, DoLa and DoLa-static have their own advantages depending on different use cases. DoLa provides another trade-off option for better balancing the cost of validation and inference.

Under potential future work, the authors may want to consider using an auxiliary model to detect when the next token is expected to be something factual (names, dates, etc) and only use DoLa decoding in these cases. This might help avoid over-triggering leading to the repetition issue described in section 2.2.

Thanks for the insightful suggestion! We believe that this is a promising direction. Contrasting layers only when it is necessary for the factual-related tokens can better balance factuality with linguistic proficiency in decoding. We will try to explore more on this idea further after this version of DoLa. We believe that the current findings in DoLa can serve as pioneering research in this direction to inspire future work to make LLMs more factual. We appreciate this insightful feedback and see it as a valuable contribution to improve DoLa in future iterations.

At the bottom of page 2 you say $j \in \\{0, \ldots, N-1 \\}$ but later on you say that the subset from which $j$ is selected is only a subset of the layers $0$ through $N-1$, referred to as $J$. Therefore should this line not instead say something like: $j \in J$, where $J \subset\\{0, \ldots, N-1\\}$?

Thanks for pointing this out! We have revised the equation according to your suggestion!

Typos:

All the typos are fixed. Thanks again for carefully reading our paper!

We thank Reviewer uyVJ for the constructive comments!

How the method performs if all previous layers are considered at each sampling step? Explanation of why they restrict the candidate layers considered to only a subset of the non-final layers.

The performance of TruthfulQA of "DoLa - all layers" is shown below. We can observe that using all layers can still improve the scores compared to the Vanilla baseline, but the improvement is not as significant as DoLa. Thus, we infer that the essential information to be contrasted is located in a certain part of the layers. Our bucket-based selection can help to find a suitable bucket within 2~4 validation test. Using all layers can also improve in most cases but it is not as significant as DoLa.

“The motivation for selecting the layer with the highest distance d(·, ·) as the premature layer is to maximize the difference between the mature/premature layers.” - explanation is tautological, you're saying you pick the largest difference because it maximizes the difference

Thanks for pointing this out! We have modified the sentence as “The motivation for selecting the layer with the highest distance d(·, ·) as the premature layer is to ensure that the model would significantly change its output after that selected layer, and thus may have a higher chance to include more factual knowledge that does not exist in the early layers before it.”

“DoLa-static has the drawbacks of 1) large search space in layers” - there are only 10s of layers to consider, in what sense is this a large search space?

Thanks for pointing this out! We have modified this claim of large search space. We agree that for smaller models like llama-7b/13b, the search space is not very large. But indeed, DoLa-static has a larger search space than DoLa. We suggest that if the user has the resource for 10s of validation runs for a specific validation set, DoLa-static is still a good option. However, if the specific validation set is not available, DoLa has a better ability to transfer to different data distributions, while DoLa-static is more sensitive to selected layers, based on our observation in Section 4.1.

Thank you once again for your insightful review! Since the discussion period will end in less than two days, we wanted to make sure that we have adequately addressed the issues you raised. We also hope that our additional experiments have addressed most of your concerns. We would appreciate your feedback on our responses!