Forcing Diffuse Distributions out of Language Models

Thank you for your detailed read of our paper.

The method is standard fine-tuning

The reviewer is correct that our method basically involves fine-tuning on samples from the desired distribution, and we will qualify our claims and framing to highlight the similarity to standard fine-tuning. Our method’s interestingness comes from the fact that fine-tuning on tasks like uniform dice rolls leads to generalization to other tasks, such as picking a random baby name. It wasn’t obvious to us ahead of time that we would see this sort of generalization, and we think it is a pretty interesting result to share with the community.

Missing fine-tuning details

Thanks for pointing out this omission. We have open-sourced our codebase and data, so all our experiments are directly reproducible from there. Here is a link to an anonymized repo, and we will link to the Github repo in the final paper. We will also add training details in the paper’s appendix.

Instruction-tuned language models

Thanks for pointing out that we were using “language model” to refer to instruction-tuned language models. We will clarify this in the paper. As a note, though we don’t cover it in our paper, our claims about non-random outputs distributions do hold pre-trained language models. For example, when GPT-2 is prompted with “She rolled the 20-sided dice. It landed on the number”, the output distribution is very non-uniform over the valid options. While we didn’t finetune any pre-trained models, we expect our method would also work well in these settings.

[W1] The tasks considered are relatively simple

We agree with the reviewer that our current method can’t be applied to improving diversity of arbitrary open-ended generation. This is an area we plan to explore more in future papers, and we will elaborate further on this limitation in our Conclusion section.

[W2] Missing details on the fine-tuning setup

Thanks for pointing out the missing information. We fine-tune all models for 50 optimization steps over all prompts, and early stop when the training loss is within 20% of the optimal loss (entropy of the target distribution). Each run takes less than 10 minutes of wall-clock time on a single A100 GPU. We will add these details to Section 3.2. We have already open-sourced all code and data needed to reproduce experiments in the paper. Here is a link to an anonymized repo, and we will link to the Github repo in the final paper.

[W3] Missing detail on tasks (Sec 5)

Since the output spaces of many of the tasks (e.g., baby names) are quite large, we couldn’t report the full empirical distribution. However, we did report metrics computed over them (coverage and entropy in Appendix B.2). Let us know if you believe there are other results that we should report.

Thank you for your detailed read and insightful comments.

Direct operations on token probabilities

It’s indeed possible to add logit biases in tasks such as random number generation to introduce randomness, but this approach fails when crafting such a candidate set is arduous (e.g., baby names) or impossible (e.g., biography generation). We will clarify that the main goal of the paper isn’t to improve generation diversity over finite candidate sets, but to elicit generalization over tasks where candidate sets are difficult to enumerate.

A sufficiently high temperature can indeed smooth out LM distributions, but this is hard to make work in practice, because of just how skewed these distributions are. For example, we computed that to match our results in Section 4.2, language models need to be sampled at a temperature of >10.

Why does our fine-tuning generalize

We are also curious about the reason behind the observed generalization of LM fine-tuning, which is an ongoing research topic that’s beyond the scope of our work. 1 offers one explanation for strong empirical generalizations of LoRA fine-tuning.

Why doesn’t fine-tuning change the model’s general capability?

We are using a rank of 4 when training the LoRA modules, so the amount of new information injected is small. We speculate that the limited expressivity of LoRA tuning means we make “local” updates to the model and shouldn’t affect general capabilities.

More prompting baselines

We acknowledge that prompt engineering, such as asking for multiple random numbers all at once could improve diversity, but as a user interacting with the system, it is unnatural to ask for a list of many numbers, and then pick one from this list. We would also like to clarify that the key motivation of fine-tuning language models to be diverse data generators is precisely to remove the need for prompt engineering and manually controlling categorical attributes.

Future directions

There are two practical applications of this approach that we are thinking of. The strong generalization ability of our fine-tuning method seems to enable diverse and high-quality data generation without human intervention. Another promising direction is de-biasing language models. Since our technique is a distribution-matching method, which lends itself naturally to de-biasing.

Contextualizing our work in literature on generation diversity

We agree with the reviewer that the diversity (or lack thereof) in LLM generations is a challenge for doing good evaluation. However, improving LM evaluation practices was not the purpose of our paper. Rather, our aim was to introduce a method that improves model usability—supporting greater diversity over valid outputs without the need for prompt hacking or decoding strategy finagling. In contrast with past literature that mostly examines linguistic (e.g., n-gram) diversity 12, we focus instead on distributional diversity (i.e., diffuseness) of language models. Our work may have impacts on how people do open-ended text generation evaluation, but improving evaluation via matching human linguistic diversity (as MAUVE does) was not our main goal.

Cross entropy is not new

Our paper proposes a view of language model training as distribution matching, as opposed to maximum likelihood estimation. Yes, we are using cross entropy, but not in the standard way of maximizing likelihood of the true next token. We are instead training to maximize entropy over the items in $\mathcal{T}$, and we show that this procedure, though it resembles standard conditional language model training in that a cross entropy term is employed, actually leads to the minimization of KL divergence with a target distribution. Thus, our method takes existing mathematical tools and employs them in a novel way. We will add an extra sentence to our methods section to make our contribution more clear.

Experiment does not provide meaningful results

We disagree with the reviewer that our experiment does not provide meaningful results. We first show that on random number generation and baby name generation, our fine-tuning technique results in models that produce distributions that are practically indistinguishable from that of a uniform distribution. While these two generation tasks are indeed somewhat contrived, results on the biography generation task show that our method improves generation diversity in practical generation settings, without requiring task-specific training.

A small note on reviewer Jrar's concerns about how evaluation of the diversity of generations are handled in a more principled way through, e.g., MAUVE.

This is a reasonable take, but I disagree and I'd like to see this paper accepted despite it not really engaging with MAUVE / measurement of diversity of generation of natural language text. Measurement of diversity of textual distributions in general is quite hard, and MAUVE in particular seems to mainly leverage the clusterings provided by the pretrained LMs we're evaluating (Pimentel et al., 2022, Cluster-based Evaluation of Automatically Generated Text). In places where there is potentially some categorical reference distribution (die rolls, countries, etc.) methods like MAUVE are unlikely to work (in particular because the clusters induced by LM embeddings aren't the natural way of, e.g., splitting the 6 possible die rolls), and either way are overkill, since one can just measure KL-divergence with the reference distribution of die rolls. Not only that, but given that the distribution of, e.g., textual descriptions of die rolls on the internet, are unlikely to be distributed according to the true distribution of die rolls, it makes sense to me to both train for and evaluate for such distributions specifically.