On the Entropy Calibration of Language Models

Thanks for the useful review! We appreciate that you highlight quality and originality as strengths of the work. From our understanding, the main weakness you highlight is that the practical implications are unclear, and you also raise specific clarification questions about diversity, calibration, and memorization. We address these points below, starting with conceptual clarifications, then addressing practical implications and additional experiments, and finally discussing other questions.

Conceptual clarifications:

(a) The first question is about what diversity means, and whether the paper uses a precise definition. Thanks for bringing up this point, which we will clarify in the revision. The first point to note is that we take diversity to be a property of the model, not individual generations. We use the definition put forth in Hashimoto et al. (2019), which is that a model’s diversity is its log loss on reference generations. The intuition behind this definition is that log loss (also known as cross entropy or forward KL) is a coverage metric: to attain low log loss, the model must “cover” as much as the reference distribution as possible. Please see Hashimoto et al. (2019) for more intuition and experiments for this definition.

(b) The review also asks about what calibration means qualitatively. The answer to this question depends on the type of miscalibration that the original model suffered from. In our paper, we show that base LLMs suffer from entropy blowup, a type of miscalibration where the entropy grows over the length of a generation. Calibrating the model then means that the generations remain coherent rather than becoming incoherent as more tokens are generated. We can see this in Appendix D, which contains randomly chosen high, medium, and low entropy excerpts of model generations. Qualitatively, we find that low entropy excerpts are either repetitive or contain verbatim copies of the training set, medium entropy excerpts are high quality, and high entropy excerpts are incoherent.

(c) Finally, the review asks about the connection between memorization and calibration. The review touches on two distinct memorization-related phenomena, which we disentangle here. First, in Appendix D, we find that model outputs containing memorized training examples have low entropy, which makes sense. Nonetheless, the tendency of models to memorize examples verbatim is outweighed by their tendency to accumulate errors, so on average they are still miscalibrated upward, with entropy too high. The second connection to memorization, as mentioned in Section 3 of the paper, is that models have the capacity to remember rare tokens seen during training. This notion of memorization is less about memorizing long excerpts verbatim, and more about the model putting nontrivial mass on rare tokens. In contrast with long memorized excerpts, which drive entropy down, outputting rare tokens instead causes the entropy to increase in subsequent steps, where the model must take in these rare tokens as context.

Practical implications:

Before addressing each point individually, we first provide a high-level discussion of the paper’s impact. The main challenge of entropy calibration is that one must control sequence-level entropy for a per-step autoregressive model that can accumulate errors. This mismatch, also known as “teacher forcing” or “exposure bias,” has been the subject of a long line of empirical papers, but remains unresolved and continues to harm generation quality (see, e.g., Williams & Zipser (1989), Ranzato et al. (2016), Welleck et al. (2020), and Welleck et al. (2024)). Framing this empirical question in terms of entropy and log loss gives us a quantitative handle on the problem, letting us make principled statements about a long-standing problem in language modeling.

In particular, in the field, many take for granted that to improve generation quality and mitigate exposure bias, one must sacrifice diversity. All existing sampling-time adjustments perform this tradeoff in different ways (e.g. temperature, top-k, top-p, min-p). By framing quality and diversity as calibration and log loss, we prove that this tradeoff is not fundamental: it is provably possible to calibrate while preserving log loss. This result opens the door to future empirical work which seeks to avoid the quality-diversity tradeoff, while also providing both a principled conceptual framework through which to evaluate potential methods, and an idealized algorithm (future entropy scaling) to aim for and approximate. This is in addition to the paper’s other main contribution, that of studying miscalibration scaling, which has direct implications for practice in that it predicts which modalities will see reduced error accumulation as models scale.

Next, we address specific questions individually.

(a) First, the review asks for experiments which ground the calibration gap. In Figure 5 of Appendix E, we plot the MAUVE for excerpts of model generations against the entropy of those excerpts, where MAUVE is an automatic text quality and diversity metric. Across all datasets and models tested, we find that MAUVE vs entropy follows an upside-down U shape, with the entropy sweet spot (i.e. the highest MAUVE) roughly coinciding with the model’s log loss. This experiment corroborates findings in past works that entropy miscalibration leads to worse outputs (see, e.g., Basu et al. (2021)).

(b) Next, the review asks about empirical experiments for the proposed theoretical algorithm of future entropy scaling. To answer this question, we run an additional experiment to provide empirical support for entropy lookahead. To make the experiment feasible, we limit the entropy lookahead to one step (Braverman et al., 2020), and we compare it to temperature scaling, which is equivalent to zero-step entropy lookahead. Specifically, we run Llama-1B on the MBPP dataset while computing the one-step future entropy for the top 32 candidate tokens for each step. We find that it attains a better log loss for every entropy target, providing practical evidence that lookahead improves the quality-diversity tradeoff (see the table below).

Other questions:

Q1: How were model and dataset size were controlled? A1: This is a great question which we will clarify in the revision. For three out of the four model families (all but Pythia), the training details are not public. On the other hand, the Pythia models were all trained on the exact same data. If the other model families were trained to be Chinchilla-optimal, then model and dataset size should be scaled proportionally. Notably, in all four model families, the scaling behavior of miscalibration is predictable, despite the fact that their training details differ. Nonetheless, they do have different scaling exponents, perhaps due to the fact that recent models are trained with higher quality data.

Q2: Could future entropy scaling be approximated by a beam-search-like procedure? A2: Regarding the specific suggestion in the review, to reweight beam candidates by estimated future entropy, if we are given a future entropy estimator then we can just do future entropy scaling and beam search is unnecessary. One other reasonable suggestion is to reweight beam candidates by the entropy so far. This is an interesting approach, and can be thought of as trying to approximate global temperature scaling with importance reweighting. For the approximation to be close, the beam would have to be exponentially large, but smaller beams might still offer improvements over vanilla sampling (albeit with increased cost).

Q3: How do pretraining dataset size and diversity affect the calibration gap? A3: This question is interesting and certainly worth exploring empirically. Section 3 of the paper provides theoretical predictions for this exact question, if we define diversity to be the dataset’s power law exponent. Specifically, the theory predicts that the calibration gap will scale as $m^{1/\alpha - 1}$, where $m$ is dataset size and $\alpha$ is the power law exponent. In other words, making the dataset larger or less diverse should decrease the calibration gap.

Thanks for the useful review! We appreciate that you found the paper to be well-framed, the research questions to be well-posed, and the experiments to be useful. From our understanding, the main weakness you highlight is that the practical takeaways are unclear. To address this concern, we describe some practical implications of the paper below.

(a) On the practical significance of entropy calibration: the main challenge of entropy calibration is that one must control sequence-level entropy for a per-step autoregressive model that can accumulate errors. This mismatch, also known as “teacher forcing” or “exposure bias,” has been the subject of a long line of empirical papers, but remains unresolved and continues to harm generation quality (see, e.g., Williams & Zipser (1989), Ranzato et al. (2016), Welleck et al. (2020), and Welleck et al. (2024)). Framing this empirical question in terms of entropy and log loss gives us a quantitative handle on the problem, letting us make principled statements about a long-standing problem in language modeling.

(b) Practical implications of future entropy scaling: in the field, many take for granted that to improve generation quality, one must sacrifice diversity. All existing sampling-time adjustments perform this tradeoff in different ways (e.g. temperature, top-k, top-p, min-p). By framing quality and diversity as calibration and log loss, we prove that this tradeoff is not fundamental: it is provably possible to calibrate while preserving log loss. This result opens the door to future empirical work which seeks to avoid the quality-diversity tradeoff, while also providing both a principled conceptual framework through which to evaluate potential methods, and an idealized algorithm (future entropy scaling) to aim for and approximate.

To provide empirical support for entropy lookahead, we also run additional experiments where we evaluate the quality-diversity tradeoff for one-step entropy lookahead (Braverman et al., 2020) and compare it to temperature scaling, which is equivalent to zero-step entropy lookahead. Specifically, we run Llama-1B on the MBPP dataset while computing the one-step future entropy for the top 32 candidate tokens for each step. This procedure is impractical in that it involves processing 32 additional tokens for every single token produced, making it roughly 33x slower than vanilla sampling. Nonetheless, we find that it attains a better log loss for every entropy target, providing practical evidence that lookahead improves the quality-diversity tradeoff (see the table below).

(c) Practical implications of the analysis: it is a priori unclear whether error accumulation is fundamental, especially as models continue to scale and saturate previously difficult benchmarks. Should we continue searching for an algorithm which addresses exposure bias, or will it go away with scale? Our principled framing lets us make the following statements, supported by both theory and experiments: (1) miscalibration improves very slowly with scale, and (2) the scaling rate depends on the heavy-tailedness of the data. These statements provide practical guidance: for example, they suggest that addressing exposure bias for creative writing is more pressing than addressing it for code, as the latter has faster miscalibration scaling.

Other questions:

(1) Additional intuition for the toy model: this is a great question which we will devote more space to addressing in the revision. A related setup which might provide additional intuition is that of n-gram models with backup: when generation begins, the model has low entropy because the prefixes are familiar and its n-gram statistics are well-populated. However, once it generates a rare token, it must back up to its (n-1)-gram statistics because its n-gram statistics are no longer well-populated, leading to higher entropy and a worse outputs. LLMs may have similar behavior, where they must back up to coarser representations when a rare token or concept is generated, increasing entropy. The toy model studied in the paper is then just a simplified / stylized version of this setup.

(2) On whether log loss is always the best calibration target: this is also a great question which we will address in the revision. Recall that for base LLMs, entropy initially matches log loss but increases as more tokens are generated (Figure 3). This entropy growth is almost always detrimental for any long-form generation task, as it is indicative of error accumulation (see, e.g., Holtzmann et al., 2020, Basu et al., 2021). Reducing sampling temperature until the entropy curve is flat then leads to entropy matching log loss; in other words, the log loss is roughly the highest entropy we can attain while not having error accumulation. This analysis suggests that base LLMs should at least be calibrated down to the log loss to make generations stable.

In terms of calibrating past the log loss, the specific entropy target likely depends on the setting. For settings where both quality and diversity matter, we provide evidence that the log loss is a reasonable target: in Figure 5 of Appendix E of our paper, we plot MAUVE (an automatic text quality and diversity metric) for excerpts of model generations against the entropy of those excerpts. Across all datasets and models tested, we find that MAUVE vs entropy follows an upside-down U shape, with the entropy sweet spot (i.e. the highest MAUVE) roughly coinciding with the model’s log loss. On the other hand, for settings where we don’t care about diversity, like one-shot question answering, it might be better to drop entropy even lower, which is often done in practice.

Thanks for the helpful review! We appreciate that you find the theoretical contribution elegant and interesting, and the empirical analysis thorough. The main weaknesses mentioned involve specific questions about the theory, which we believe result from minor misunderstandings that we will clarify in the revision.

The first question asks about a formal analysis of one-step lookahead. This a great question which we will devote space to addressing in the revision. One-step lookahead does not attain full sequence calibration, but our analysis can be easily adapted to this case because we can simply replace the future entropy predictor with the one-step entropy, and treat one-step entropy as an alternative approximator for the full future entropy. The $\delta$ term in Theorem 5.2 then becomes the expected error between the one-step and full future entropies. This analysis suggests that as we increase the number of lookahead steps, the approximation error should improve, making $\delta$ smaller and smaller.

The second question asks whether Assumption 5.1 is reasonable, if it involves controlling the test error of a recursively defined quantity. Actually, one key contribution of our algorithm is that it avoids this recursion, so Assumption 5.1 does not involve any recursively defined quantity. Predicting the future entropy of $\hat p^{(ent)}_{\alpha}$ would indeed involve predicting a recursively defined quantity. Instead, our algorithm makes two key changes: (1) it replaces each future entropy term with a fixed function $f_t$ defined in previous iterations of the algorithm, and (2) it iterates backward from $T$ to $1$, so all subsequent entropy terms have already been fixed. As a result, each future entropy predictor $f_t$ just needs to fit a fixed target function, whose values are bounded $[0, (T-t) \log V]$, with no recursion required. We will clarify this point in the revision.

Other questions:

1. On using $\hat p$ to approximate recursive future entropy: this would be interesting to try. However, it doesn’t improve much upon Algorithm 1 in terms of practicality, which is already polynomial time, because it still involves sampling a huge number of sequences when performing calibration. One would need to sample roughly $T \log V$ continuations per candidate token per step to approximate future entropy with constant error, so we would need to process $V T^2 \log V$ additional tokens for every 1 token generated. Instead, we run an additional experiment where we replace $\hat p^{(ent)}_{\alpha}$ with $\hat p$ as you suggested, and we also limit the lookahead to just one step so that the experiment is feasible (i.e. Algorithm 2 in Braverman et al., 2020). Specifically, we run Llama-1B on the MBPP dataset while computing the one-step future entropy for the top 32 candidate tokens for each step, which is 33x slower than vanilla sampling. The resulting algorithm attains a better quality-diversity tradeoff than temperature scaling, which is equivalent to zero-step entropy lookahead, providing empirical support for the effectiveness entropy lookahead. However, is still insufficient to avoid the tradeoff entirely, suggesting that more steps of lookahead are likely necessary to further improve the tradeoff. Thanks for the suggestion!

2. On why the newer model families attain better scaling exponents: this is a very interesting question. Unfortunately, the specific training details of three of the four model families (all but Pythia) are not publicly available. If we were to speculate, one of the main changes from past to current training practices is that newer models are pre-trained in multiple stages, with later stages (“midtraining”) involving higher quality data and specific domains of interest (like code). The pretraining data mixtures are also likely very different. Based on our analysis in Section 3, the data distribution has a large effect on how quickly miscalibration improves with scale, so changes in the pre-training data and data schedule are very likely to alter the calibration scaling curves. On the other hand, the architectures for these models are publicly available, and are very similar across model generations, suggesting that they are unlikely to be the cause.

Thanks for the helpful review! We appreciate that you found the paper well-written and recognized its core theoretical and empirical contributions. From our understanding, the main weakness you highlight is that the practical implications are unclear, which we address below.

Before addressing each point individually, we first provide a high-level discussion of the paper’s impact. In the field, many take for granted that to improve generation quality, one must sacrifice diversity. All existing sampling-time adjustments perform this tradeoff in different ways (e.g. temperature, top-k, top-p, min-p). By framing quality and diversity as calibration and log loss, we prove that this tradeoff is not fundamental: it is provably possible to calibrate while preserving log loss. This result opens the door to future empirical work which seeks to avoid the quality-diversity tradeoff, while also providing both a principled conceptual framework through which to evaluate potential methods, and an idealized algorithm (future entropy scaling) to aim for and approximate. This is in addition to the paper’s other main contribution, that of studying miscalibration scaling, which has direct implications for practice in that it predicts which modalities will see reduced error accumulation as models scale.

Next, we answer each question individually.

The first question is whether miscalibration always hurts downstream performance, given that some tasks like creative writing may benefit from higher entropy. The answer depends on the type of miscalibration: if a model is miscalibrated in that its entropy grows over the length of a generation, that is almost always detrimental for any long-form generation task, as it is indicative of error accumulation (see, e.g., Holtzmann et al., 2020, Basu et al., 2021). This is the form of miscalibration that even the largest base LLMs suffer from: entropy initially matches log loss but increases as more tokens are generated (Figure 3). Reducing sampling temperature until the entropy curve is flat then leads to entropy matching log loss; in other words, the log loss is roughly the highest entropy we can attain while not having error accumulation. This analysis suggests that base LLMs should at least be calibrated down to the log loss to make generations stable.

If by miscalibration we mean the exact entropy target to maximize downstream performance, the specific calibration target likely depends on the setting. For settings where both quality and diversity matter, our paper provides evidence that the log loss is a reasonable target: in Figure 5 of Appendix E of our paper, we plot MAUVE (an automatic text quality and diversity metric) for excerpts of model generations against the entropy of those excerpts. Across all datasets and models tested, we find that MAUVE vs entropy follows an upside-down U shape, with the entropy sweet spot (i.e. the highest MAUVE) roughly coinciding with the model’s log loss. On the other hand, for settings where we don’t care about diversity, like one-shot question answering, it might be better to drop entropy even lower, which is often done in practice.

The second question is about the practical takeaways from the proposed theoretical algorithm, future entropy scaling. First, as discussed above, the theory proves that one can avoid the quality-diversity tradeoff, which has immense practical implications for settings like test time compute scaling and synthetic data. To provide empirical support for entropy lookahead, we also run additional experiments where we evaluate the quality-diversity tradeoff for one-step entropy lookahead (Braverman et al., 2020) and compare it to temperature scaling, which is equivalent to zero-step entropy lookahead. Specifically, we run Llama-1B on the MBPP dataset while computing the one-step future entropy for the top 32 candidate tokens for each step to adjust their logits. This procedure is impractical in that it involves processing 32 additional tokens for every single token produced, making it roughly 33x slower than vanilla sampling. Nonetheless, we find that it attains a better log loss for every entropy target, providing practical evidence that lookahead improves the quality-diversity tradeoff (see the table below).