The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning

Base model capabilities The reviewer raised a great point on the impact of the choice of the pretrained model on the performance improvements achieved through entropy minimization. We observe that the improvements from EM-FT and EM-RL on Llama-3.1-8B-Instruct are less substantial on math reasoning tasks compared to those on Qwen-2.5 (Sec. 5). The intuition behind using entropy minimization is based on the observation that models tend to be more accurate on examples for which they make predictions with higher confidence [4,5,6]. However, [7] reveal that entropy minimization based methods improves performance as long as the finetuning data align closely with the training data. As the fine-tuning data diverges from the training distribution, the model's performance diminishes. Building on this observation, we conjecture that entropy minimization is most beneficial when there is reasonable alignment between the pre-training and fine-tuning data.

A common assumption in RL, and test time scaling for LLMs is that it is expected that the base models can already solve a problem in N trials, i.e., we require at least one successful trajectory to reinforce and compute the advantage [2]. In output-verified RL, this assumption is applied by using accuracy based filters during training, which filters out un-solvable problems based on the success rate over N trials. This hypothesis suggests that the base model must generate high-quality responses with reasonable probability and coverage over the downstream task. Extrapolating this assumption, entropy minimization via EM-RL, EM-FT and EM-INF sharpens the probability distribution over all responses based on the model's confidence. We conjecture that this sharpening mechanism is expected to be more useful on base models which show better scaling under best of N [8]. Table 1 (rebuttal) shows the percentage gain under best of n sampling (N=4) compared to greedy decoding (N=1) averaged over MATH, AMC, Minerva and OlympiadBench. We observe that Qwen-2.5-math-7b shows larger gains compared to LLaMA-3.1-8B-instruct when number of responses is increased from N=1 to N=4, indicating that Qwen is more likely to produce a correct solution. We hypothesize that better inference time scaling (e.g. best of n) is a likely cause for improvements from EM-RL [8].

Table 1: Average performance on MATH, AMC, Minerva and OlympiadBench with greedy decoding (N=1) and best of N (N=4). We select the best response by using self-consistency, i.e., we pick the answer with the highest frequency

To further understand initial model choice, we finetune another base model- DeepSeek-Math-7B-Instruct [11] on the same data and training settings as Qwen-2.5-7B-Math (Sec. 3.3). Deepseek-7b-math-instruct has been finetuned on math-related tokens for 500B tokens on top of Deepseek-coder-7b by the model creators. We use the instruction tuned version which has been further finetuned using chat data. We fine tune deepseek-7b-math-instruct using entropy minimization (EM-RL) and evaluate it in Table 2 (rebuttal), we observe that entropy minimization improves performance over the Deepseek-math-7b-instruct by 2.5% on average on math tasks. Compared to LlaMA both, Qwen-2.5-math and DeepSeek-7b-math-instruct have undergone domain specific finetuning [11,12]. This observation suggests that gains from EM-RL are not only limited to Qwen, and can be seen on stronger models tuned for the downstream task.

Table 2: Performance comparison of unsupervised EM-RL against RLOO for DeepSeek-Math-7B-Instruct. We use the setup defined in Sec. 3.3. Bold indicates performance improvement over RLOO.

The success and limitations of our method highlight the importance of the capabilities of the pretrained models. We call for the inclusion of EM as a baseline in the evaluation of future post-pretraining and inference-time scaling algorithms, to better separate the contribution of algorithmic innovations from the intrinsic capabilities of the model itself (Line 58). We will update Sec.3.4 with these analyses and results to better contextualize model choice. We will also rephrase (Line 48, Sec. 6. Conclusion) to better acknowledge the impact of initial checkpoints on RL tuning.

Entropy minimization and Supervised RL The reviewer raises an interesting parallel of combining entropy minimization with supervised RL. [3] shows that typical maximum entropy reinforcement learning techniques (e.g. PPO, GRPO, Soft actor-critic) which apply entropy maximization struggle on challenging control tasks where a precise subset of actions are required, and the remaining actions will lead to non-recoverable states and consequently low final rewards. They find that maximum entropy regularization can encourage the policy to select multiple similarly mediocre actions rather than the best one for short-term entropy benefits. Extrapolating this observation, reasoning tasks such as math and coding, require a precise sequence of actions to arrive at the correct solution. We conjecture that entropy minimization (EM-RL) along with supervised RL can encourage the policy to take the most precise action over the several mediocre choices.

Furthermore, [1] find that supervised RL finetuning in its default setting (RLOO/GRPO) drops entropy during the early stages of training. Meanwhile, the policy’s validation performance presents an inverse trend, where it rises rapidly when training starts, and then saturates at a certain level. They find that entropy needs to be effectively maintained during training to avoid saturation of policy performance, as also observed in [11]. This observation suggests that, for a given performance level, a policy with higher entropy is preferred, since low entropy policies are difficult to further improve. Building on these observations, we conjecture that if the pre-trained model has converged and is at a higher entropy level, further entropy minimization through EM-RL can help as a post-training step, since there is more entropy to consume by EM-RL. We will add this insight to Sec. 3.4 of the paper.

Entropy maximization vs minimization:

Thank you for pointing out this interesting work! Similar to [9], we tried to maximize entropy, but observed that entropy maximization alone without the policy loss collapses the policy after 10 gradient steps while using Qwen-2.5-7B-Math, this observation was also noted in [9]. While [9] observes performance improvements, we conjecture this is because they train on a single carefully selected example. Furthermore, [9] trains performed on test-set examples where making the model more random (larger entropy) could be desirable, similar to best of n [2]. We find that only negative entropy reward (entropy minimization) is sufficient for RL with similar effects without requiring any data selection. Standalone entropy maximization does not converge and eventually collapses, but EM-RL and EM-FT via entropy minimization converge and can be used to train on large datasets (60K prompts) without output verification. We will add this discussion to Sec.3 of the paper.

Code release We will release the code for reproducibility. Since the rebuttal guidelines do not allow external links, we will attach it to our final version.

[1] Cui, Ganqu, Yuchen Zhang, Jiacheng Chen, et al. The entropy mechanism of reinforcement learning for reasoning language models

[2] Beirami, Ahmad, Alekh Agarwal, et al. "Theoretical guarantees on the best-of-n alignment policy

[3] Zhang, Ruipeng, Ya-Chien Chang, and Sicun Gao. When Maximum Entropy Misleads Policy Optimization

[4] Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization.

[5] Subhankar Roy, Aliaksandr Siarohin, et al. Unsupervised domain adaptation using feature-whitening and consensus loss.

[6] Wang, D., Shelhamer, E., Liu, S., et al, T. Tent: Fully test-time adaptation by entropy minimization.

[7] Press, Ori, Ravid Shwartz-Ziv, et al. The entropy enigma: Success and failure of entropy minimization

[8] Huang, Audrey, Adam Block, et al Self-improvement in language models: The sharpening mechanism

[9] Wang, Yiping, Qing Yang, Zhiyuan Zeng, et al. Reinforcement learning for reasoning in large language models with one training example

[10] Yang, An, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu et al. Qwen3 technical report.

[11] Shao, Zhihong, Peiyi Wang et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models

[12] Yang, An, Beichen Zhang, et al. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement

Impact of initial model choice

The reviewer raised a great point on the impact of the choice of the pretrained model on the performance improvements achieved through entropy minimization. We observe that the improvements from EM-FT and EM-RL on Llama-3.1-8B-Instruct are less substantial on math reasoning tasks compared to those on Qwen-2.5 (Sec. 5). The intuition behind using EM is based on the observation that models tend to be more accurate on examples for which they make predictions with higher confidence [6,7,8] (L19, L76). However, [9] reveals that EM based methods improves performance as long as the finetuning data align closely with the pretraining data. As this fine-tuning data diverges from the training distribution, the model's performance diminishes. Building on this observation, we conjecture that EM is most beneficial when there is reasonable alignment between the pre-training and fine-tuning data.

A common assumption in RL, and test time scaling for LLMs is that it is expected that the base models can already solve a problem in N trials, i.e., we require at least one successful trajectory to reinforce and compute the advantage [10]. In output-verified RL, this assumption is applied by using accuracy based filters during training, which filters out un-solvable problems based on the success rate over N trials [11,12]. This hypothesis suggests that the base model must generate high-quality responses with reasonable probability and coverage over the downstream task. Extrapolating this assumption, entropy minimization via EM-RL, EM-FT and EM-INF sharpens the probability distribution over all responses based on the model's confidence [13]. We conjecture that this sharpening mechanism is expected to be more useful on base models which show better scaling under best of N. Table 1 (rebuttal) shows the percentage gain under best of n sampling (N=4) compared to greedy decoding (N=1) averaged over MATH, AMC, Minerva and OlympiadBench. We observe that Qwen-2.5-math-7b shows larger gains compared to LLaMA-3.1-8B-instruct when the number of responses is increased from N=1 to N=4, indicating that Qwen-2.5 is more likely to produce a correct solution. We hypothesize that better inference time scaling (e.g. best of n) is a likely cause for improvements from EM-RL, since we observe larger improvements on Qwen2.5 compared to LLaMA on math (Sec.5).

Table 1: Average performance on MATH, AMC, Minerva and OlympiadBench with greedy decoding (N=1) and best of N (N=4). We select the best response by using self-consistency, i.e., we pick the answer with the highest frequency.

To further understand initial model choice, we finetune another base model- DeepSeek-Math-7B-Instruct on the same data and training settings as Qwen-2.5-7B-Math (Sec.3.3). Deepseek-7b-math-instruct has been finetuned on math-related tokens for 500B tokens on top of Deepseek-coder-7b by the model creators [1]. We use the instruction tuned version which has been further finetuned using chat data. We fine tune deepseek-7b-math-instruct using entropy minimization (EM-RL) and evaluate it in Table 2 (rebuttal), we observe that EM improves performance over the Deepseek model by 2.5% on average on math tasks. Compared to LlaMA both, Qwen-2.5-math and DeepSeek-7b-math have undergone domain specific math finetuning [2,1]. This observation suggests that gains from EM-RL are not only limited to Qwen, and can be seen on stronger models finetuned for the downstream task.

Table 1: Performance comparison of unsupervised EM-RL against supervised RLOO. Bold indicates performance improvement over RLOO. We use the same training setup defined in Sec. 3.3.

The success and limitations of our method highlight the importance of the capabilities of the pretrained models. We call for the inclusion of EM as a baseline in the evaluation of future post-pretraining and inference-time scaling algorithms, to better separate the contribution of algorithmic innovations from the intrinsic capabilities of the model itself (Line 58). We are unable to analyze the pre-training data of either LLaMA and Qwen models, since they have not been released and acknowledge that studying the impact of base model choice in RL is of great importance. We will update Sec.3.4 with these analyses and results to better contextualize model choice. We will also rephrase (Line 48, Sec. 6. Conclusion) to better acknowledge the impact of initial checkpoints on RL tuning.

Evaluation on knowledge intensive tasks We evaluate EM-INF and EM-RL on knowledge intensive tasks- UGPhysics and MMLU and observe that EM can improve performance UGPhysics, but maintains the same performance on MMLU.

UGPhysics is a question answering benchmark for Physics which requires undergrad level physics knowledge [3]. We observe that entropy minimization with logit optimization (EM-INF) improves Qwen-2.5-7B-Instruct, Qwen-2.5-math-7B-Instruct, and Llama-3.1-8B-Instruct by 4% on average, indicating that unsupervised EM-INF helps improve pre-training performance by making the model more deterministic (Table 3). We will add this description about UGPhysics in Sec.4 to clarify.

We additionally evaluate MMLU-stem-COT [4] performance for our RL tuned models (EM-RL) in Table 3 (rebuttal). We observe that EM-RL did not impact its performance on knowledge intensive MMLU.

Table 3: Performance of EM-RL compared with GRPO and RLOO on the MMLU-stem-COT dataset [4].

Sensitivity to KL regularization coefficient $\beta$

We provide results with different $beta$ for EM-RL token below in Table 4 (rebuttal). We observe that a small $\beta$ value is better for entropy minimization because it weighs EM more in the overall loss function (L172).

Theoretically, the KL regularization can increase the entropy when the base model is too uncertain, i.e., it deviates from the pre-trained checkpoint causing it to visit high entropy states. However when $\beta<1$ (we use $0.001$) (L172) the dominant gradient in the policy gradient is from entropy minimization. Since we set $\beta$ to a small value the overall objective minimizes the entropy.

Table 4: Sensitivity of EM-RL-token to various values of the KL regularization coefficient.

What is j in Formula 5 Thank you for pointing this out! $j$ indexes over the vocabulary, i.e., $j$ ranges from 1 to the vocabulary size. We will correct the notation to $L_{EM-INF} = max (-\sum_{j=1}^{|{V}|}\sigma(z_t)_j\log \sigma(z_t)_j,\delta)$

COT example

We provide example generations with entropy minimization at inference time (EM-INF) via logit optimizations in the Appendix (Figure 4). For EM-INF, we observe that the model produces more deterministic and concise outputs, since entropy minimization optimizes the model to seek a specific mode. This behaviour helps improve on complex/highly uncertain tasks such as scientific coding.

We provide EM-RL generations below and compare it with regular RLOO model generations. We observe that both EM-RL and RLOO invoke the self-verification step in their generations. These observations indicate that the entropy minimization makes the model more deterministic and generates a more concise outputs while preserving self-verification. We will add these qualitative analysis to the results section.

RLOO [omit] Let's check if this solution works. The five integers are 1, 8, 10, 10. [omit] The median is 2 greater than the arithmetic mean, [omit] So, the solution works. The least possible value for the mode is 10.

EM-RL-seq [omit] We should verify that this solution works by checking the values of a, b, c, d and e The median is [omit] 2 greater than the median. [omit] So, the solution is correct. Thus, the least possible value for the mode is 10.

[1] Shao, Zhihong, Peiyi Wang et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models

[2] Yang, An, Beichen Zhang, et al. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement

[3] Xu, Xin, Qiyun Xu, et al Ugphysics: A comprehensive benchmark for undergraduate physics reasoning with large language models

[4] Hendrycks, Dan, Collin Burns, et al. Measuring massive multitask language understanding

[5] MMLU-Stem-COT subset from Deepseek evaluator: https://github.com/deepseek-ai/DeepSeek-Math

[6] Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization

[7] Subhankar Roy, Aliaksandr Siarohin, et al. Unsupervised domain adaptation using feature-whitening and consensus loss

[8] Wang, D., Shelhamer, E., Liu, S., et al, T. Tent: Fully test-time adaptation by entropy minimization

[9] Press, Ori, Ravid Shwartz-Ziv, et al. The entropy enigma: Success and failure of entropy minimization

[10] Beirami, Ahmad, Alekh Agarwal, et al. Theoretical guarantees on the best-of-n alignment policy

[11] Yu, Qiying, Zheng Zhang, et al. Dapo: An open-source llm reinforcement learning system at scale

[12] Cui, Ganqu, Lifan Yuan, et al. Process reinforcement through implicit rewards

[13] Huang, Audrey, Adam Block, et al Self-improvement in language models: The sharpening mechanism

Error bars

We thank the reviewer for the great suggestion. We provide all result tables with error bars below by running the evaluation three times with a different random seed. We report the mean $\pm$ standard deviation for all models. We observe that our trend is still consistent with EM-RL, without any labeled data, achieving comparable or better performance than RL baselines such as GRPO and RLOO that are trained on 60K labeled examples. We will add these error bars to aid clarity and reproducibility in the paper. Furthermore, our trends with inference time logit optimization (EM-INF) is still consistent with improving the performance of the base model for almost all tasks and model classes by 3% on average.

Table 1: Performance comparison of unsupervised finetuning (EM-FT) and various rewarding methods in EM-RL with supervised finetuning and RL. Italics, Bold indicates performance improvement over GRPO and SC-RL (self-consistency RL), respectively.

Table 2:Performance comparison of EM-INF against test-time scaling methods. Bold & Italics indicates the best and second best performance, respectively. Dash line ("–") denotes that selfconsistency is inapplicable. We use temperature, t= 0.1 and sample N = 1 trajectory unless specified

Sensitivity to number of rollouts $N$

The reviewer has a great observation that the trajectory level entropy estimator has a high variance. We set $N=4$ building on existing RL fine tuning works to make sure that the training overhead is consistent [1]. We measure the impact of the number of rollouts, N on EM-RL-sequence in Table 4 (rebuttal). We observe that EM-RL-sequence benefits with larger $N$ on MATH-500, suggesting that larger number of rollouts might reduce the variance. However, we note that a larger N does not help AMC, Minerva and Olympiad bench, and reduces performance on AIME. Overall, we observe consistent performance of EM-RL-sequence on sweeping N from 4 to 16. We will add these results to Sec. 3.4 of the paper as a plot.

Table 4: Sensitivity to number of rollouts N of EM-RL-sequence. We use Qwen-2.5-Math-base and use the training setup defined in Sec. 3.3.

Effectiveness of adaptive temperature

Thank you for this suggestion. We will update Sec.4 results by adding the following: Adapting the temperature (adaptive temperature) at every decoding step to reduce the entropy boosts model performance by 3% on average. Adaptive temperature is a simple method which is broadly applicable to all tasks while being computationally efficient.

Formal proof for prop 1

Thank you for pointing this out! A more formal proof is provided below. We will add this to Sec. 4.

Proposition: Both adaptive temperature scaling and logit optimization reduce the entropy of a model’s output distribution. However, in high-uncertainty settings (i.e., when the model’s predictive distribution has high entropy), logit optimization can change the order of non-top logits. In contrast, temperature scaling preserves the order of logits and merely sharpens or flattens the distribution proportionally. Both temperature scaling and logit optimization will keep the token with the top logit and increase its probability, making them less likely to change the model’s behavior when it is highly confident.

Proof: Part 1: $\forall **z** \in \mathbb{R}^{|\mathcal{V}|}$ (logits) let $p(**z**) = \text{softmax}(**z**)$. Let $\mathcal{H}(p)$ denote the entropy of $p$; $\mathcal{H}=-\sum_{i=1}^{|\mathcal{V}|}p_i \log p_i$. $\exists **z**$, $\eta>0$ and $a,b\in[1,\dots,|V|]$, such that $**z**_a<**z**_b$ and $**y**_a>**y**_b$ where $**y**_i = z_i + \eta p_i(\log p_i + \mathcal{H})$, $i=1,\dots,|\mathcal{V}|$.

Consider $\mathbf{y_a}-\mathbf{y_b} = \mathbf{z_a}-\mathbf{z_b} + \eta (g_a-g_b)$ where $g_i = p_i (\log p_i + \mathcal{H})$. $g$ is the gradient of entropy $\mathcal{H}$ w.r.t. $\mathbf{z}$.

We now show that $\exists \mathbf{z}, a,b$ such that $z_a<z_b$ and $g_a>g_b$.

By Mean-Value Theorem $\exists c \in (p_a,p_b)$ such that $g_a-g_b = (p_a-p_b)(\log c + \mathcal{H} + 1)$. When $p_a-p_b < e^{-(\mathcal{H}+1)}$ we have

$c<p_b<e^{-(\mathcal{H} + 1)} \implies \log c + \mathcal{H} + 1 < 0 \implies g_a-g_b > 0$

Thus, we can set an arbitrary large $\eta$ such that, $z_a-z_b + \eta(g_a-g_b) > 0$

Part 2: $b$ cannot be $\arg \max_{i} z_i$.

Proof: Let $j = \arg\max_i \mathbf{z_i}$, i.e. the index of the maximum logit. Since $p_j = \max_i p_i$ we can bound it using the inequality:

$\mathcal{H} (p) = -\sum_i p_i \log p_i \geq -\log p_j \quad \Rightarrow \quad p_j \geq e^{-\mathcal{H}(p)}\geq e^{-(\mathcal{H} (p)+1)}$

We now analyze how $g_i$ behaves as a function of $p_i$. Let $g(p_i) = p_i (\log p_i + \mathcal{H})$ and $g'(p_i) = \log p_i + \mathcal{H} + 1$. Thus, $g(p_i)$ is monotonically increasing whenever $g'(p_i)>0$, i.e. when, $p_i > e^{-(\mathcal{H} + 1)}$ So no lower-probability index $i$ can have $g_i>g_j$, i.e., the entropy gradient cannot reduce $\mathbf{y_j}$ relative to others. Therefore, $b\neq j$, leading to the following behavior

$0<g_a\leq g_b \quad p_a \in (e^{-\mathcal{H}},p_b]$

$g_a < 0 < g_b \quad p_a \in (0,e^{-\mathcal{H}})$

Decoding method for sampling?

We use sampling based decoding with a temperature of 0.1 for Table 2. We will add this to the Table caption to aid reproducibility.

l162-4: this sentence is unclear to me. What does “lost in the sauce” mean?

Thank you for pointing this out. We will rephrase it as the following to make it more formal- “​​For example, problems requiring a complex, long chain of thought, where being more deterministic at every step helps prevent the model from losing track of the reasoning process.”

Is the KL regularization term used for both EM-RL-token and EM-RL-sequence or just the former?

The KL term is used in both EM-RL-token and EM-RL-sequence. We will add this to Sec. 3.2 more explicitly.

[1] Cui, Ganqu, Lifan Yuan, et al. Process reinforcement through implicit rewards

When is entropy minimization helpful?

The reviewer raised a great point on when EM works. The intuition behind using EM is based on the observation that models tend to be more accurate on examples for which they make predictions with higher confidence (L19, L76). However, [1] reveals that entropy minimization based methods improves performance as long as the finetuning data align closely with the pretraining data. As this fine-tuning data diverges from the training distribution, the model's performance diminishes. Building on this observation, we conjecture that EM is most beneficial when there is reasonable alignment between the pre-training and fine-tuning data, as supported by model performance on downstream tasks.

Furthermore, a common assumption in RL, and test time scaling for LLMs is that it is expected that the base models can already solve a problem in N trials, i.e., we require at least one successful trajectory to reinforce and compute the advantage [2]. In output-verified RL, this assumption is applied by using accuracy based filters during training, which filters out un-solvable problems based on the success rate over N trials. This hypothesis suggests that the base model must generate high-quality responses with reasonable probability and coverage over the downstream task. Extrapolating this assumption, EM-RL, EM-FT and EM-INF sharpens the probability distribution over all responses based on the model's confidence [4]. We conjecture that this sharpening mechanism is expected to be more useful on base models which show better scaling under best of N.

We will update Sec.3.4 with this discussion to better contextualize when entropy minimization can help. We will also add this to our future work and limitations section and acknowledge that the success and limitations of entropy minimization is a great research direction.

Over-confidence and calibration

Thank you for raising this interesting point on over-confidence and calibration! Entropy minimization (EM-RL, EM-FT) encourages the model to produce low-entropy (i.e., highly confident) predictions. We conjecture that entropy minimization can both improve or hurt calibration. If entropy minimization helps the model become more confident in the right direction- that is, it becomes more confident on correctly generated solutions, then both performance and calibration improves. We expect that when the pre-trained model’s data distribution closely aligns with the fine-tuning task and the pre-trained model produces the correct response with a reasonable probability, entropy minimization should benefit performance and calibration. On the other hand, entropy minimization can harm calibration when it reinforces highly confident (low entropy) incorrect solutions to be generated with higher probability. We measure the expected calibration error (ECE) of both LLaMA and Qwen 2.5 models in Table 2 (rebuttal). We observe that the ECE for Qwen models improves, likely because EM-RL substantially benefits Qwen 2.5 math on math reasoning (Sec.3.4) . On the other hand, we notice that entropy minimization harms LLaMA’s calibration likely because we noticed smaller improvements on using EM-RL with LLaMA. We conjecture that the impact of entropy minimization on calibration is tied to the base model capability and expect that entropy minimization should likely benefit or have minimal impact on model performance and calibration.

Table 2: Expected Calibration Error for EM-RL-sequence and EM-RL-token compared to the base model on the AMC math dataset. Lower ECE is better.

Non-reasoning tasks We evaluate EM-INF and EM-RL on knowledge intensive tasks- UGPhysics and MMLU and observe that EM can improve performance UGPhysics, but maintains the same performance on MMLU.

UGPhysics is a question answering benchmark for Physics which requires undergrad level physics knowledge [3]. We observe that entropy minimization with logit optimization (EM-INF) improves Qwen-2.5-7B-Instruct, Qwen-2.5-math-7B-Instruct, and Llama-3.1-8B-Instruct by 4% on average, indicating that unsupervised EM-INF helps improve pre-training performance by making the model more deterministic (Table 3). We will add this description about UGPhysics in Sec.4 to clarify.

We additionally evaluate MMLU-stem-COT performance for our RL tuned models (EM-RL) in Table 3 (rebuttal). We observe that EM-RL did not impact its performance on knowledge intensive MMLU. This observation along with calibration errors, indicates that entropy minimization does not necessarily make the model too confident on non-reasoning tasks.

Table 3: Performance of EM-RL compared with GRPO and RLOO on the MMLU-stem-COT dataset.

Entropy maximization during Pre-training Entropy maximization during pre-training is an interesting idea to explore! [5] find that supervised RL finetuning in its default setting (e.g. GRPO) drops entropy during the early stages of training. Meanwhile, the policy’s validation performance presents an inverse trend, where it rises rapidly when training starts, and then saturates at a certain level. They find that entropy needs to be effectively maintained during training to avoid saturation of policy performance. This observation suggests that, for a given performance level, a policy with higher entropy is preferred, since low entropy policies are difficult to further improve. Building on these observations, we conjecture that if the pre-trained model has converged and is at a higher entropy level, further entropy minimization through EM-RL can help as a post-training step can benefit, since there is more entropy to consume by EM-RL. We will add this insight to Sec. 3.4 of the paper.

Logit optimization

EM-INF treats the model’s output logits as if they were free parameters, and uses gradient descent to update them to minimize the entropy of the distribution they induce; no gradient wrt. or the model parameters or parameter update is needed (Line 260). This optimization is performed during each decoding step to reduce the entropy of the LLM’s distribution without any parameter update. After logit optimization, we use sampling based decoding to select the next token. EM-INF can change the order of non-top logits (proposition 1), thereby generating different sequences. Empirically, we find that EM-INF works on complex tasks with high uncertainty (e.g. AIME math, UGPhysics and SciCode)(Table 3). By using EM-INF, we observe that Qwen 32B can outperform frontier models like GPT-4o on Scicode and is 3x more efficient than inference scaling through self-consistency and sequential refinement (Table 4, Figure 1).

[Tab. 2] Does 60k unlabeled prompts mean the prompts from the particular task just without the provided answers?

Yes, the 60k unlabeled prompts is a collection of prompts without the ground truth answers.

Self-consistency on long output tasks

Thank you for pointing out this interesting experiment! We agree that self-consistency can be applied using more sophisticated measures of consistency beyond exact matching, such as, fuzzy string match, embedding based similarity and using another LLM to measure consistency. We leave experimentation with these techniques as future work and we will clarify this throughout our paper to acknowledge this.

We experiment with fuzzy matching for self-consistency for EM-INF in the Table 4. We find that our trend that entropy minimization (EM-INF, adaptive temperature) improves inference time scaling methods (self-consistency and iterative refinement) by 3% still holds. We will add these results to our paper and acknowledge that self-consistency can be performed on long outputs using methods like fuzzy string matching.

Table 4: Performance comparison of EM-INF against test-time scaling methods on LeetCode. Bold & Italics indicates the best and second best performance, respectively. We use temperature, t= 0.1 and sample N = 1 trajectory unless specified. For self-consistency on long outputs, we pick the output which has the lowest average Levenshtein distance to all other outputs.

"1e-6" 1e-6 means 1*10^-6. We will replace L198 with this expression for clarity.

Typo and language Thank you for pointing these out! We will fix the typo and replace "lost in the sauce” with: For example, problems requiring a complex, long chain of thought, where being more deterministic at every step helps prevent the model from losing track of the reasoning process.”

[1] Press, Ori, Ravid Shwartz-Ziv, et al. The entropy enigma: Success and failure of entropy minimization

[2] Beirami, Ahmad, Alekh Agarwal, et al. Theoretical guarantees on the best-of-n alignment policy

[3] Xu, Xin, Qiyun Xu, Tong Xiao, et al Ugphysics: A comprehensive benchmark for undergraduate physics reasoning with large language models

[4] Huang, Audrey, Adam Block, et al Self-improvement in language models: The sharpening mechanism

[5] Cui, Ganqu, Yuchen Zhang et al. The entropy mechanism of reinforcement learning for reasoning language models.