We appreciate the thoughtful and enthusiastic review.

Temperature comparison:

We also found the temperature scaling comparisons “insightful” and emphasize the importance of finding general purpose approaches to improving output variability orthogonal to temperature scaling which trades off prompt adherence and diversity.

Choice of Language Models:

Open-source models. At the time of selection, we chose Llama 3.1 as it was the state-of-the-art open-weight model. We will include Qwen-3 as well in the final revision as it has grown in popularity and capability. We do not anticipate significant differences in results as new open-source models get released since the proprietary models also show the same limitation. Particularly, we anticipate mode-collapse in instruction-tuned models may become even more severe in reasoning models where models have been trained on objective single-answer math problems. We leave analysis of reasoning focused models as future work.

Proprietary models. We chose OpenAI’s 4o models as and claude-3.5-sonnet as they demonstrate the key general domain capabilities characteristic of large language models, while remaining affordable for extensive resampling experiments. As models are becoming cheaper to experiment with, we are happy to include more models in the evaluation if desired.

Intuitive presentation

We are glad the reviewer finds the approach intuitive with the carefully chosen examples we used to illustrate the approach. We believe the intuitive simplicity of SimpleStrat makes it easy to incorporate in other settings.

We appreciate the thoughtful review and the constructive feedback! We hope the following clarifications and experiments address your questions.

Limitations of Self Prompting:

It’s indeed the case that the approach is sensitive to the quality of the stratification. In fact, our ablation in Figure 4 indicates the 20-questions prompting-approach is crucial to the quality of the stratification and subsequently the resulting diversity.

It’s certainly true that stratification can be discovered with other techniques. We are familiar with (Hu et al): https://arxiv.org/pdf/2505.05145 which analyzes the contribution of the attention heads to the final prediction, but the semantic analysis is limited to simple tasks. It’s not immediately obvious if the head-wise distinction would always lead to semantically interpretable strata (possibly better with SAE). Is this what you mean by leveraging activations? We would be curious to understand better how neuron activation and intermediate representations can be used to identify semantic strata as future work.

We added an additional ablation to emphasize the importance of quality partitions by only changing the 20 questions:

CoverageQA Results with std and additional Ablations -- Original Figure 4

We note that in addition to relying on quality partitions. The inclusion of external variance introduced by sampling constraints is also critical. We see SinglePrompt approach where we append the prompts together and allow the model in prompt space to select the positive or negative constraint lead to signification regression even compared to baseline GPT-4o. Noting that the language model has been trained to ignore random irrelevant tokens. The key to self-stratification is producing token efficient constraints with outsized impact on model's diversity. Critically, these small number of constraints allowing the system to inject external randomness to token space giving $2^n$ candidate prompts.

Computational overhead:

As articulated in section 3.1, we advocate for caching of AutoStratification and Heuristic Estimation per prompt. Effectively, once the stratification has been discovered, it can be reused and the excess token overhead viewed as amortized cost across many rounds of resampling. For a cached prompt, only a small overhead of ~20 tokens per constraint (3-5 constraints per problem on average) is needed for each query in Probabilistic Prompting.

Under extreme latency conditions, the model can be assumed to be small (1B-8B) with latency below 10ms to the first token. This should render the minimal per query overhead to be negligible. In highly-resource constrained and latency-sensitive settings, we can go further to optimize latency by precomputing stratification for common prompts.

CoverageQA realism:

We agree there is some risk of a sim-to-real gap between the Wikipedia CoverageQA questions/answers and more natural human generated questions/answers. We have taken several measures to address this:

1. As discussed in line 247, we manually curate the answer set to ensure the answers are reasonable. Although the heavy-lifting has been done by the WikiData community. This manual check was crucial in both removing poorly phrased or ambiguous questions but more importantly identifying missing answers. For instance, certain noble gasses were missing from the WikiData queries. We are also careful to not over punish the model by enumerating possible ways of expressing the same answer – for instance countries (e.g. "United Arab Emirates","UAE", and "Emirates") as well as for names (e.g. "J. Robert Oppenheimer", "Robert Oppenheimer", "Oppenheimer", and "J. R. Oppenheimer").

2. We separately show evaluation on CoverageQA-curated in Appendix B. This set represents a small set of 15 questions generated by the authors: e.g. give me a NFC East team. Although at a smaller scale, this evaluation gives us some confidence the wiki generated questions are representative of real questions. SimpleStrat shows similar improvement in that setting.

3. WritingPrompts experiments show diversity improvement on prompts purely human generated on Reddit. Although diversity is measured differently given the open-endedness of the prompts, this gives us confidence that SimpleStrat discovers productive stratification.

Although these measures have been taken, we agree human annotation and pair-wise comparison like LMArena can be added to further improve the quality of the CoverageQA dataset and potentially broaden the scope of questions to more subjective domains.

Thanks for reading our rebuttal. As a minor clarification, we do not create 20 stratification but rather prompt (Appendix E) the model to propose 5 candidate first questions to ask in "20 questions" the game. This game requires players to try to narrow down the "hidden" item within 20 questions. As such the questions at any point of the game should eliminate half of the solutions space.

Measuring diversity is a difficult open problem. As such, we believe the settings we introduce of creative writing and questions answering in the setting of multiple answers is a good balance of complexity while allowing for concrete measures of diversity. This said, we are open to evaluating on additional settings if you have suggestions.

We appreciate your overall positive reception and agree that diversity in language modeling "remains relatively underexplored compared to, e.g., diffusion models where this is a very active area of research." We are excited to see where this research direction leads.

We appreciate the thoughtful review and the constructive feedback! We hope the following clarifications and experiments address your questions.

Figure 4 Legend:

Anthropic stands for Claude-3.5 sonnet. This is in contrast to the Baseline which stands for GPT-4o which is a more direct comparison since the instantiation of SimpleStrat we implement uses GPT-4o. SS stands for SimpleStrat, SS Ablation stands for the Ablation described above where we replace the 20-questions approach to AutoStratification with a direct prompt asking for partitions. Thanks for raising bringing this confusion to our attention; we will rename the manuscript's legend to be clear and consistent with the main text's names.

Ablation:

The existing ablation coined SS Ablation in the main result in Figure 4 constitutes replacing both using a direct prompt instead of the 20-questions inspired prompting in AutoStratification and heuristic estimation is replaced with uniform sampling instead of heuristic estimation.

In addition to the ablations provided in the original manuscript, we add two additional ablations: as suggested, we consider the setting where we provide all the instructions to the model asking it to generate both the partitions and the sampling probabilities in one step. Second, to analyze the importance of the 20-questions approach to AS, we substitute only that prompt with a simple direct prompt asking for partitions.

We rename SS Ablated from the manuscript to AttrPrompt to emphasize the similarity of the ablation to AttrPrompt and the role it serves as a baseline approach. We add two additional ablation requested by other reviewers:

• SimpleStrat w/o 20-Questions ablates the 20 questions approach keeping everything else the same
• SimpleStrat Single Prompt ablates the need for multiple stages. We append the prompts for the three stages into a single stage and give the model a 1000 token prompt and a budget of 1000 tokens to follow the instructions. We chose to only sample temperature 0 and temperature 1 as the results show substantially worse performance than directly prompting gpt-4o.

The ablations further support the claim that 20-Questions formulation leads to far better partitions and is crucial to the observed improvements. The single prompt ablation supports the importance of having multiple stages. In particular, the single prompt is missing probabilistic prompting severely limiting the diversity. The standard deviation gives us confidence that the gains of full SimpleStrat are not the result of sampling variance.

Baseline to increase sample diversity:

We disagree that no baselines aimed at sample diversity were included. We argue temperature scaling is the most common and widely accepted baseline to increasing sample diversity. We show just applying SimpleStrat to temperature zero gives as much benefits to diversity as increasing the temperature to 1.2 in CoverageQA and 1 in WritingPrompts. We not only show absolute improvement over temperature scaling but orthogonal gains that show even at temperature 1.5 we can improve the recall on CoverageQA from 0.35 to 0.45. We further compare to AttrPrompt (Yu et al 2023) labeled SS Ablation which we believe to be the most comparable baseline also introducing variance by sampling from a family of prompts.

Zhu et al. [1] and Garces Arias et al.,[2] will naturally shine in settings with longer generations as it selectively chooses where to spend “temperature” based on model confidence. In the short answer setting of CoverageQA, these approaches are similar to just temperature scaling as there are few (5-10) tokens in the generation. Moreover, as illustrated in Figure 1 and Figure 7, the instruction-tuned models tend to be poorly calibrated in terms of next-token probability. In the context of the WritingPrompts experiment, the goal is to highlight the degradation of quality at higher temperatures. Thus, SimpleStrat accesses a different kind of variance than accessible using model confidence.

We discuss LMX[3] in line 127 of the related works. This approach relies on carefully choosing a few-shot examples from a seed set of diverse examples to extract variance. Our approach does not rely on external guidance by providing few-shot examples. A natural extension of SimpleStrat to consider is “sampling without replacement” where past samples can be leveraged to generate new unique samples. In the “without replacement” setting, SimpleStrat can be extended in 3 key ways:

1. AutoStratification can be used to introduce partitions that distinguish already seen solutions from not yet seen solutions,
2. AutoStratification can repartition the remaining unobserved solution space,
3. Probabilistic Prompting can upweighting partitions that have produced less than expected samples. In this setting LMX and SimpleStrat can be viewed as complementary mutators for discovering new solutions. In summary, LMX cannot be directly compared to SimpleStrat as it relies on a strong seed bank of solutions.

We appreciate the detailed review and the constructive feedback! We hope the following clarifications and experiments address your questions.

Major Questions

We designate answer to major questions with (Q#)

On code availability and dataset availability (Q7):

As part of the submission, we included a zip supplemental file including the experiment pipeline as well as plotting scripts. We include the prompts in the appendix of the manuscript for easier access and legibility. Additionally, we provided the CoverageQA datasets and raw inference outputs from the language models in this zip file. In particular, we want to highlight that the CoverageQA evaluation pipeline requires careful handling of redundant ways of expressing the same answer – for instance countries (e.g. "United Arab Emirates","UAE", and "Emirates") as well as for names (e.g. "J. Robert Oppenheimer", "Robert Oppenheimer", "Oppenheimer", and "J. R. Oppenheimer"). Our inclusion of the experiment pipeline in the supplemental is important for replicability given the need for handling these duplicates.

In summary, CoverageQA has already been made available in the submission. As noted by the reviewer, CoverageQA is procedurally generated and thus will be periodically updated with future versions based on changes to WikiData.

Experiment robustness (Q1-Q3),

As recommended, we reran the results in figure 4 and 5 to obtain statistical confidence. In the original experiment, we focused our compute/cost budget on running a wider variety of temperature and large number of resampling to provide confidence that the results are not due to randomness of samples. As requested, below are the results of replicating the figure 4 CoverageQA experiment on a smaller subset of temperatures (to reduce cost) repeated 5 times. They confirm the results are robust to sampling:

CoverageQA Results with std and additional Ablations -- Original Figure 4 (Q1)

We rename SS Ablated from the manuscript to AttrPrompt to emphasize the similarity of the ablation to AttrPrompt and the role it serves as a baseline approach. We add two additional ablation requested by other reviewers:

• SimpleStrat w/o 20-Questions ablates the 20 questions approach keeping everything else the same
• SimpleStrat Single Prompt ablates the need for multiple stages

The ablations further support the claim that 20-Questions formulation leads to far better partitions and is crucial to the observed improvements. The single prompt ablation supports the importance having multiple stages. In particular, the single prompt is missing probabilistic prompting severely limiting the diversity. The standard deviation gives us confidence that the gains of full SimpleStrat are not the result of sampling variance.

WritingPrompts Results with std -- Original Figure 5 (Q2)

As recommended, we report the standard deviation of the metrics over the 100 samples for each temperature setting. The generations require producing full outlines of stories based on creative writing prompts from reddit, WritingPrompt. This reported variance ensures against sampling variance over the set of prompts taken from the full writing prompts dataset.

Baseline GPT-4o

SimpleStrat with GPT-4o

We emphasize that SimpleStrat is able to randomize in the prompt space even for temperature zero. As seen the diversity as measured by cosine distance in the embedding space is significantly larger than without SimpleStrat and is comparable to much higher temperatures. It’s interesting to note that even with temperature = 0 on longer generation tasks the generated outlines have small deviations.

(Q3): For results on distributional diversity, the constraints are generated with temperature 0 and the next token probabilities are computed deterministically from the model weights. As such, there's no natural notion of sample variance. (Q5) We will update the checklist in the appendix to clarify which experiments we provide standard deviations for.

On open source models (Q4):

At the time of selection, we chose Llama 3.1 as it was the state-of-the-art open-weight model. We believe the learnings are general as the observation of existing mode collapse and improvements by SimpleStrat are consistent with analysis on proprietary models. We will include Qwen-3 as well in the final revision as it has grown in popularity and capability. We do not anticipate significant differences in results as new open-source models get released since the proprietary models also show the same limitation. Particularly, we anticipate mode-collapse in instruction-tuned models may become even more severe in reasoning models where models have been trained on objective single-answer math problems. We leave analysis of reasoning focused models as future work.

On limitations (Q6),

In section 3.4, We acknowledge the risk that SimpleStrat identifies “unwanted bias or unwanted factors” when proposing strata and proportions. Specifically, we mention that even in the case that the LLM correctly follows the 20 questions-inspired instruction to divide the space of unique solutions into equal proportions, this may have unintended consequences. For instance, there are more female baby names than male baby names as reported by the census. In that case, SimpleStrat would exhibit a preference for female baby names. We will include a dedicated section to emphasize the potential limitation of relying on model produced stratification in the main text.

To further elaborate, although SimpleStrat shows improvement empirically, it is sensitive to the model selecting good meaningful axes in auto-stratification and correct joint probabilities in heuristic estimation. As work on LLMs for forecasting improves, we expect LLMs to produce better estimates especially when given access to external data and data analysis tools. For our prototype, we restricted ourselves to studying the model’s intrinsic capabilities and tendencies. Further, the model may have biases concerning race and gender that may be reflected also in the auto-stratification and heuristic estimation. As such, it is recommended the probabilistic prompt distribution of SimpleStrat is carefully inspected for critical applications. Finally, CoverageQA is a dataset of short responses. Although this makes evaluation more practical, we believe SimpleStrat will be especially impactful in settings that require low temperature, especially multi-step reasoning as identified by Zhang et al. (2024).

Other questions:

SimpleStrat Ablation (SS Ablation):

To clarify, the abbreviation SS Ablation refers to the ablation described in section 5.1 under coverage diversity. The ablation asks the language model directly to propose criterion instead of the 20-questions approach in AutoStratification and the ablation applies uniform sampling akin to AttrPrompt (Yu et al 2023) instead of using heuristic estimation. We will make this explicit in the caption as recommended.

Abbreviated Legend in Figure 4:

Indeed Baseline indicates using GPT-4o without SimpleStrat, Anthropic stands for Claude-3.5 sonnet, SS stands for SimpleStrat, SS Ablation stand for the Ablation described above where we replace the 20-questions approach to AutoStratification with a direct prompt asking for partitions. We will make the recommended changes to the manuscript make sure the legend is clear and precise.

Anthropic Model Temperatures:

Footnote 3 for figure 4 explains that Claude models are only allowed temperatures between 0 and 1; the footnote unfortunately rendered on page 9. This is possibly because alignment of models is more difficult to guarantee for high temperatures.

Blue-Seven Phenomena:

We will include a citation for blue-seven and raise a caveat that the effects may vary across cultures: Simon, William E. "Number and color responses of some college students: Preliminary evidence for a “blue seven phenomenon”." Perceptual and motor skills 33.2 (1971): 373-374.

Dear Authors,

Thank you for addressing my comments. Please ensure that the promised revisions are incorporated into the final version of the paper - please pay particular attention to reporting statistical significance and explicitly discuss limitations in the main text. Conditional on the faithful implementation of these changes, I have raised the quality score from 2 to 3 and the overall ranking from 4 to 5.

Thanks for revisiting the review! We will implement the changes discussed in the rebuttal we described and your detailed comments on smaller changes to make the manuscript clearer.