Base Models Beat Aligned Models at Randomness and Creativity

We thank the reviewer for finding that our work “expands knowledge of LLMs” and is “very clearly written”! We address questions and comments below:

restriction of experiments to single model family

At the request of multiple reviewers, we have carried out experiments on Qwen-2.5 models. At roughly comparable sizes, we see similar results to Llama, for example, seeing more than 10X increase in divergence after alignment on random integer generation (see Table A below). We note that the base Qwen performance is significantly worse than base Llama, but the effect of alignment seems the same. We will incorporate results on Qwen in the final version, and are happy to add other models as resources allow.

| size   |   Base |   Instruct |
|:-------|-------:|-----------:|
| 7B     |    792 |      10970 |
| 72B    |    744 |      15000 |

Table A: Divergence ($\chi^2$) of random integer sampling with Qwen 2.5 models (directly comparable to Figure 2 in the paper, lower is better)

confusing sentence “… that may breakdown… ” and typos

Thank you for noticing this typo — this is indeed supposed to be “break down”, and the intended meaning is that the common intuition of improvement through scale also breaks down in some of our experiments.

Thank you for noticing the other typos in our work. These (and a number of typos we also noticed later) will be fixed in our final version!

We thank the reviewer for recognizing that our work “addresses a critical yet underexplored topic”. We address comments and questions below:

Proxies vs Real-world

We agree that our tasks do not reflect all complex aspects of the real world. We made this design choice for 2 reasons. First, such tasks tend to incorporate many different capabilities (unpredictability, coherence, world knowledge, etc.) making it difficult to isolate and evaluate individual aspects such as unpredictability. Using proxy tasks allows us to define settings with one central capability and clear evaluation metrics (such as $\chi^2$). Second, given the failure of strong aligned models on even these simple tasks, it seems unlikely that they can be effectively unpredictable in more complex settings. As abilities improve, more complex evaluations will be required, yet simple tasks serve as an effective first hurdle.

Task-appropriate randomness

We thank the reviewer for pointing out that more detail would be useful in this part of the work—we are happy to add a discussion of ε-equilibrium. For example, in the case of rock paper scissors, expected payoff for a uniform strategy is 0 points regardless of the adversary. All models achieve a payoff of at most -15 points, meaning all have significant incentive to unilaterally deviate to a uniform strategy (≥15), indicating non-ε-equilibrium for small ε. We are happy to expand on this however it might be useful.

Human evaluation concerns

First, we note that the scale of the study, including pilots, already accounts for 100s of dollars at a reasonable pay rate, which is a limiting factor. We will still attempt to increase the scale by a reasonable amount.

We have calculated Cohen's Kappa with ~70 additional comparisons, finding 0.33, 0.27, and 0.67 for originality, pleasantness, and overall preference (respectively). Our original goal in this evaluation was to use contests to find consensus under possible disagreement, given the subjective nature of the task and past findings that agreement as a notion of quality overlooks genuine sources of disagreement [2, 3, 4]. We also note that we included one control question in each job, comparing a generated poem to one composed of randomly selected words from other poems. Annotators chose the generated poem in the vast majority of cases. We will include the full templates, results of control experiments, and agreement information in the final version.

Error analysis and causes

We are happy to include more examples and analysis of error cases, such as repetitive poems generated by aligned models (see examples below), comparing these effects across model sizes and alignment recipes. We will also extend our analysis of model biases in games, e.g., building on our observation in Figure 4 that aligned models tend to become more confident following ties/wins vs. losses. We will include new experiments studying the factors affecting breakdown of randomness. For example, we run random letter generation (Table B below, similar to Figure 2 but generating letters “a” to “k”) and find that aligned models still collapse onto peaks, but the top letters differ between alignment recipes and model sizes. This suggests that collapse onto single answers is not simply the result of very likely tokens, as the preference of models for “7” may have suggested.

Entropy analysis concerns, and “Could decoding strategies … recover lost randomness?”

First, we would like to clarify that all sampling parameters (outside of Appendix A.3.1) are set for pure sampling i.e. no truncation (top-p/k) and a temperature of 1.0. We aimed to directly compare distributions, avoiding confounding factors. The reviewer is correct that sampling parameters can have an effect, although out of top-p/k and temperature, only temperature would increase randomness/entropy. Appendix A.3.1 suggests that even relatively high values of temperature do not allow aligned models to match randomness of base. In the limiting case (very high temperature), aligned models would become uniform but also incoherent, once all tokens have the same probability. We will include the full results of our temperature grid, where even very high temperatures (which greatly decrease the quality of aligned models) failed to match the uniformity of base models.

Longer form tasks

Although we have not explicitly tested this, there is likely a family of long-form tasks (e.g. artistic/creative storytelling) where both base and aligned models will often fail—aligned models will not be original enough, while base models will not be structured enough. We believe there is an exciting space of hybrid solutions that combine base and aligned models, although this is beyond the scope of our work here. Overall, this highlights the fact that more complex tasks make it harder to isolate individual capabilities for evaluation, one motivation for keeping our tasks relatively simple.

We thank the reviewer for positive comments! We are happy to address any concerns as they arise.

We thank the reviewer for positive comments and careful feedback. We respond to comments and questions below:

different model families such as Qwen

As requested, we ran the Qwen model family, and broadly see results in line with our previous findings. For example, for random integer generation at comparable sizes to Llama (Table A below), Qwen also sees a large jump in divergence after alignment, despite the base model already being significantly more divergent than Llama. We will include results for Qwen in the final version and are happy to add other models as resources allow.

| size   |   Base |   Instruct |
|:-------|-------:|-----------:|
| 7B     |    792 |      10970 |
| 72B    |    744 |      15000 |

Table A: Divergence ($\chi^2$) of random integer sampling with Qwen 2.5 models (directly comparable to Figure 2 in the paper, lower is better)

Games with Longer Time Horizons

We agree that these games (such as chess and Go) would be interesting. The main challenge is that the role of randomness is much less clear/harder to evaluate. For example, chess endgames (the final few moves) can often be solved totally deterministically by the winner, and therefore unpredictability may actually be a negative. A main goal in our work was providing explicit evaluations in settings where the benefit of unpredictability is clear, and so this may be out of our scope. However, we believe a study of when and how models may use unpredictability in these games would be extremely valuable.

How to quantify the impact of pretraining data on the pattern of generations?

A first step in understanding the impact of pretraining data is studying models with different pretraining. For example, added experiments on Qwen models (Table A, above) show that pretraining may have an impact on initial unpredictability, as Qwen base is significantly more divergent from uniform than similarly-sized Llama. This may indicate that the inclusion of synthetic data during pretraining, as used in Qwen, has a significant effect.

Tulu-SFT is consistently second best

While a definitive explanation will need more evidence, we have some ideas for why this may happen. First, this may relate to post-training objectives. Other aligned models explicitly incorporate reward with methods like RLHF and DPO, which could push models towards a highest-reward output and cause peaks. In contrast, SFT uses the same cross-entropy objective as pretraining which works to learn the underlying data distribution. At a higher level, this may be related to a general trade-off between unpredictability and popular tasks like those in the Open LLM Leaderboard. SFT does not push models as far as other alignment techniques towards solving such Leaderboards, but this may also mean it does not trade off as much ability to be unpredictable.

We are happy to hear the reviewer finds our “experimental design is compelling” and our work “addresses an important consequence of alignment”. We respond to reviewer questions and comments below:

“other base models? … Qwen”

As suggested, we replicated experiments with Qwen-2.5. Results support both our findings that alignment reduces unpredictability, and the reviewer’s idea that the Qwen base model “already exhibit[s] the same issues”. For example, in random integer generation (Table A below), base Qwen models at comparable sizes to Llama are significantly more divergent (e.g. $\chi^2$ of 744 for Qwen vs 355 for Llama at ~70B parameters). Alignment still results in a large (more than 10X) increase in divergence.

We agree that including DivPO (Lanchantin et al. 2025) would be valuable. Model weights do not seem to be available yet, but we will add this once they are. We predict that DivPO will result in performance closer to base models in Figure 1, with strong unpredictability but weaker performance than other aligned models on the OpenLLM Leaderboard.

| size   |   Base |   Instruct |
|:-------|-------:|-----------:|
| 7B     |    792 |      10970 |
| 72B    |    744 |      15000 |

Table A: Divergence ($\chi^2$) of random integer sampling with Qwen 2.5 models (directly comparable to Figure 2 in the paper, lower is better)

Novelty vs. existing works on diversity

While past work on linguistic diversity is relevant, there is not a clear optimal level of diversity that models should have, making evaluation less definitive. One way our work is distinct is in proposing tasks where success is clearly defined, e.g., total uniformity in random integer generation.

Solutions or mitigation, more demanding formats

The central goal of our work was to develop a deeper understanding and more explicit evaluation for these effects of alignment, but we can suggest possible solutions. We believe the two most promising directions are enhanced alignment techniques that more explicitly handle this tradeoff (e.g. Lanchantin et al. 2025, which you mention), and creating hybrid systems that allow the strengths of base and aligned systems to work together.

This is related to your suggestion of “more demanding creative writing formats like storytelling”. We agree that base models may fail due to a lack of coherence, while aligned models would fail due to a lack of originality. A system that allows the two to work together may solve this. We will discuss further in the updated version.

suggested citations on amplified cognitive bias

We thank the reviewer for the suggested citations. We will add these along with other citations such as [1,2], which also deal with cognitive biases and aligned models.

Does manipulating the prompt help?

While models are sensitive to prompting, this does not seem to be a solution. For random number generation, the prompt is very clear that randomness is needed, but the issue persists. For games, we did experiment with prompt versions that encode the optimal strategy, yet there tended to be a ceiling on randomness for the aligned models. Base models showed some brittleness and could be less random for certain prompts, but saw consistently better performance and a much higher ceiling. We will add this study to the final version.

Causes: Training and Model Internals

Given the consistency of this effect, we believe it is quite likely that it would be observable during post-training, likely including telltale changes in model weights. A natural next step would be pairing the evaluations we developed with iterative post-training checkpoints (e.g. OLMo 2 which has weights at every 20 training steps) to observe when and how these issues arise. We also believe that the objective function may have an important effect, given that SFT models (which use cross entropy-based probabilistic learning) seemed to fare better than models that explicitly incorporate reward. This may require a more comprehensive study, post-training multiple models with different objectives.

References

[1] Liu, Ryan et al. “Large Language Models Assume People are More Rational than We Really are.” ICLR (2025)

[2] Jones, Erik and Steinhardt, Jacob “Capturing Failures of Large Language Models via Human Cognitive Biases.” NeurIPS (2022).

We are happy that the reviewer found our work “clear and easy to understand” and our experiments “important for general knowledge”. We address comments and questions below:

“What takeaways should the reader have?”

One high-level takeaway from our work is that aligned models do not always represent the limits of current model capabilities. While they are by far the most popular, base models may succeed on tasks for which aligned models fail, which readers should consider when selecting models. A second, related takeaway is that more work is needed to understand the effects of alignment. As part of this, our work supports the idea of broader evaluations for models, especially evaluations for which single, static responses are not enough. Finally, new methods are needed to combine the benefits of aligned and base models.

Samples vs logits

When designing our experiments, our main concern with using logits was that these require picking an exact prefix to the response token, while both base and aligned models often vary generation format for a given prompt (e.g., adding “\n”, “Here is my answer:” etc. after the input prompt). Sampling allows the model to generate naturally across different formats from which answers can be extracted, while using logits would constrain the model to one format. However, recorded logits throughout, and generally saw the same patterns, meaning this distinction may not be too significant. We include a comparison in the final paper.

Role of Sampling Parameters

We use pure sampling in our main experiments (no truncation, temperature=1.0) to directly compare model distributions and avoid confounding variables. We agree that parameters would have an effect, although truncation (top-p/k) will only reduce entropy (i.e. make models even more predictable). We did experiment with temperature in Appendix A.3.1, although even when raising the aligned model entropy to match the base model, the aligned model was still more predictable at random number generation. We will include a more extensive grid of temperatures and further analysis in the final paper.

Single class of model

At the request of multiple reviewers, we produced results for the popular Qwen family of models as well, at comparable sizes to Llama. We see similar effects of alignment, for example, on random number generation (Table A below). Note that base Qwen is significantly more divergent than base Llama at similar sizes, possibly due to the synthetic task data included in pretraining. Still, alignment results in a more than 10X increase in divergence at both model sizes. We are happy to extend to more models as resources allow, and will incorporate these results in the final paper.

| size   |   Base |   Instruct |
|:-------|-------:|-----------:|
| 7B     |    792 |      10970 |
| 72B    |    744 |      15000 |

Table A: Divergence ($\chi^2$) of random integer sampling with Qwen 2.5 models (directly comparable to Figure 2 in the paper, lower is better)

Random sampling of tokens instead of numbers

We ran this experiment, asking for uniform samples of letters from “a” to “k” (analogous to Figure 2, which uses integers 0-10). We see a similar overall pattern, with alignment making divergence significantly worse. While “7” was the most likely number across models, the most likely letter depends on both model and size—all 8B Tulu models prefer “j”, while all 70B aligned models prefer “f”. This suggests that convergence towards a mode is not simply the result of an overwhelming bias across models (as “7” may have suggested). We will include these results and further analysis in the paper—thank you for the very informative suggestion!

| size   | Base      | Instruct    | Tulu DPO   | Tulu SFT   | Tulu Full   |
|:-------|:----------|:------------|:-----------|:-----------|:------------|
| 8B     | 78 ("a")  | 1158 ("e")  | 614 ("j")  | 370 ("j")  | 832 ("j")   |
| 70B    | 193 ("k") | 10513 ("f") | 1177 ("f") | 709 ("f")  | 1231 ("f")  |

Table B: Divergence ($\chi^2$) for random letter generation with Llama-based models (directly comparable to Figure 2, lower is better). Results are listed as: $\chi^2$ (most likely letter)