Generative Monoculture in Large Language Models

Thank you for your thoughtful comments on our paper.

(W-1-1): While it is unlikely that models we consider do not use the datasets we consider given the extreme breadth of data scraped from the internet to train most LLMs, we can not provide definitive proof of their presence given the closed nature of these models, and the lack of reliable dataset detection methods [1].

(W-1-2): We appreciate this point. We believe the goodreads dataset is our attempt at an approximation of a sample of the book review training data, and while there likely is some deviation from the true distribution of training data (which is unknown and impossible to determine for these models), we think there is good reason to believe our main point still stands: note just how skewed the LLM-generated book reviews are towards positive sentiment: both GPT3.5 and 4 had returned 100% of their responses rated in the most positive bucket under prompt (1), and never returned less than 97.6% in the most positive bucket (see Takeaway 1). It is hard to fathom that the overall (unknown) distribution of book reviews is this skewed, with next to no negative samples— especially given that the goodreads data is a part of the training data as confirmed above and has data in this range. This suggests that there is at least some part of the distribution that has strong negative sentiment, and since we have evidence of no strong negative sentiment generations, we believe this suggests that generative monoculture is occurring.

Impact of source filtering (W1-3) Upon review, we find that there is little correlation between a sample’s length and sentiment i.e., no biases are introduced. Please find the figure in Appendix H of the revised draft.

Impact of target filtering (W2) We filter out low-quality generated results because they were illegible (gibberish), and could not be recognized as a book review. Similar to only wanting to compare diversity on correct code solutions, we only want to compare sentiment on generations that are recognizable as a book review. We adopted the reviewer’s suggestion to analyze whether this filtering step would introduce additional bias; it turned out that the change in average sentiment score is on the magnitude of 0.001, meaning that the impact of filtering on the sentiment is negligible.

We have also provided some examples of book reviews at different perplexity scores in Appendix C.1. Since we could not manually inspect all outputs, we investigated random samples of generations at various perplexity scores and decided on a threshold that admitted the most generations possible while still being comprehensible. Another reason we looked into perplexity is to understand the practicality of high randomness generation as a mitigation—as we presented in Table 1 and explained at line 463; although high randomness could lead to improved diversity, the generation quality degraded to a point where it could not be adopted for practical use.

1. We are well aware of the existence of the work by Oren et al. However, as noted in their abstract: "the tendency for language models to memorize example order means that a contaminated language model will find certain canonical orderings to be much more likely than others. Our test flags potential contamination whenever the likelihood of a canonically ordered benchmark dataset is significantly higher than the likelihood after shuffling the examples.". Thus, this paper is also not useful in helping understand the training distribution for several reasons: (a) it works on understanding "memorization" -- which may not have happened at all (in fact, we don't see verbatim memorization in the generated outputs), (b) it is possible that the dataset originally was shuffled to begin with (their tests assume that the dataset was originally un-shuffled, and they compare this with a shuffled counterpart), and (c) their evaluation is done for models of size 1.4b, and there's no definitive guarantee that this is going to be reliable or useful for larger models. This is why we chose not to invest our limited rebuttal time in pursuing this rabbit hole.

2. It is completely perplexing for us that the reviewer thinks that GoodReads has a low probability of being included, when prior work shows far more rudimentary tabular datasets have been memorized (https://arxiv.org/pdf/2404.06209); none of the other reviewers share this concern and prior ``published'' works make similar (not the same) observations with NO knowledge of the training data (https://arxiv.org/pdf/2402.05070, https://arxiv.org/pdf/2303.17548, and https://aclanthology.org/2024.emnlp-main.244/).

3. Based on your feedback, we have considered a NEW dataset that is provably OUTSIDE the cutoff date for training these models; this means that your concern of contamination should now be resolved. We have run experiments, and shown that w.r.t this NEW dataset, monoculture exists. At this point, we fail to understand what remaining concerns the reviewer has. If the reviewer believes that this NEW dataset is also influenced somehow by the old unknown training distribution (without articulating how or why), then it's unclear what experimental evidence will satiate them -- their claim that the demonstration of monoculture being unfounded is both harsh and unreasonable. Particularly when the reviewer also acknowledges that it is virtually impossible to truly learn the training distribution, or its interplay on task performance. By this rationale, work of this nature can never be published unless large companies share their data composition, or truly open source models (like OLMo) are performant on a wide variety of tasks.

4. If everything else that we are saying is convincing, and you too likely believe that monoculture exists, why not reflect this in your score? It seems to us that the reviewer is ideologically in agreement with the phenomenon we present, appreciates the nuances in the experiments, and can't think of a better methodology than we have already presented. Yet the reviewer states that this work is not good enough?

Hey authors,

thank you for taking the time to rebuttal. I have raised my rating because: (1) the authors included an additional experiment with an accurate source distribution, providing concrete evidence of generative monoculture. Combined with the main text's findings, despite uncertainties about the source distribution, I agree that the paper successfully demonstrates the phenomenon of generative monoculture.

I remain concerned about the metric aspect due to the inaccuracy of the training distribution, which is unavoidable for most LLMs with unknown training data. In my view, the best way to address this concern is to conduct another experiment on OLMo (both pre-trained and aligned ones) with a task it can handle. So that this paper gives one measurement for monoculture from one experiment that has no concerns of inaccuracy.

I agree that Oren et al. is not particularly helpful here, as confirming the dataset is part of the training data only addresses a small portion of the issue. The source distribution can still be inaccurate. Ideally, this could only be resolved if the full training data were known.

We would first like to shed light on the following differences:

1. Setting A: The traditional training ecosystem operates as follows: There is pre-training (the model obtained only after the step is denoted PT), followed by instruction fine-tuning (IFT) and preference tuning (or alignment). This final model is called Chat. The results outside the appendix are presented for models trained in this regime.

2. Setting B: However, for the models trained in the appendix, the pipeline is as follows: there is an additional round of supervised fine-tuning (SFT) done using the new dataset on both the PT model -- resulting in PT-SFT model, and on the Chat model resulting in the Chat-SFT model. In an ideal universe, we would have liked to add the new dataset to the PT phase for the Chat models too, but can't do so. So this is a coarse grained approximation of that setup.

Having established this, please find our responses below:

1. In Setting A, we have consistently noted that in comparison to the source distribution, both PT + Chat models are less diverse. These are the results that are presented in the main body of the paper.

2. In Setting B (for figure 33a), we see that both (i) PT-SFT , and (ii) Chat-SFT models are less diverse compared to the source distribution: 3 colors in PT-SFT, Chat-SFT vs. 4 colors in source for prompt 1. In prompt 2 (figure 33b), we see that the PT-SFT model is less diverse than the source, but the Chat-SFT model is comparable.

3. Why is Chat-SFT more diverse than PT-SFT in figure 33b? Recall setting B: We perform additional fine-tuning on a previously aligned model to obtain the Chat-SFT model. We conjecture this to be the reason for increased diversity in prompt 2 (which explicitly requires the model to follow a particular persona in the instruction i.e., has a greater ``degree of instructability''). However, we would like to stress that 3 out of the 4 scenarios considered show the monoculture phenomenon clearly (reduced diversity compared to the source) in this non-standard training regime i.e., RLHF is likely to hurt diversity if it is the "last step" in model training (which is not the case for the Chat-SFT models).

We hope this has clarified your concerns. Thank you for giving us this opportunity.

Could the authors help me understand why "new, unseen dataset: We filter out books published after October 1, 2023 from the GoodReads dataset (May 2024 ver)" is "provably OUTSIDE the cutoff date for training these models"? Isn't this dataset consisting of books before October 1, 2023? Shouldn't be books after October 1, 2023 be provable "OUTSIDE the cutoff date"? Did I miss anything?

We thank the reviewer for engaging with us during the rebuttal. This ICLR, a common concern that was raised was the lack of reviewer engagement, so thank you for taking the time to respond to most of our comments. We did not want to bother you over the weekend, hence our delayed response.

We understand your sentiment, but as stated several times during the rebuttal period, your concern is not one that can be addressed in any manner, let alone one that is reasonable within the time frame. To succinctly restate our response: even in scenarios where we are fully aware of the training distribution, it is impossible to determine if this particular distribution is fully responsible for the performance on a given task (unless the model is trained on one distribution to perform one task); thus using a random/representative sample of this distribution (that we can provably show was part of the training data) is the best approximation one can make. This is what we did in our experiments as part of the submission, and what we did for the experiment in the rebuttal as well. While the reviewer agrees that they believe the monoculture phenomenon, we are writing to understand if there’s any setting/experiment they are aware of that will convincingly respond to this concern? Note: Using OLMo is not useful, as stated in our rebuttal for 2 reasons: 1. It is not great for long form generation as needed in our experiments, and 2. We can't attribute generations to specific parts of the training data.

We thank the reviewer for the comments and questions. Please find our response below.

Generation and correctness evaluation of code solutions. Let us clarify–we kept generating solutions and measuring their correctness, until at some point all problems reached at least 20 correct solutions. We then stopped. That is why we ended up generating 100 solutions per problem for GPT-4 and 200 for Claude-3–essentially, GPT-4 is better at generating correct solutions and we stopped early for it while we had to continue generating more for Claude-3. Comparing Fig. 5a with Fig. 26 in Appendix, we can see that GPT-4 does achieve higher accuracy in general compared to Claude-3.

Size of the Code dataset. a) The reason we are limited to working with “easy” problems is the following: the model generated solutions on harder questions achieved extremely low correctness/accuracy, while for the purpose of our study (on the other attributes) we focus on only the correct solutions (lack of diversity is not as important when the generated solutions themselves are incorrect) and we need at least 20 correct solutions. By our procedure (described in “Generation and correctness evaluation of code solutions”), that means we needed to generate a large number of solutions before having just a few that are correct; this experiment was very expensive (because of the large number of API calls needed). This is the inherent difficulty posed by the limited capability of the models, and we do not have better solutions before modes become more inexpensive, and get more capable at generating correct code.

b) Regarding the set of 100 problems, we wanted to clarify that this means we generated and autojudged 10,000 solutions (100 x 100) in total by GPT-4 (and 20,000 solutions in total by Claude-3) on these 100 problems (described in “Generation and correctness evaluation of code solutions”). Moreover, the results on 100 problems already revealed pretty significant trends. It is true that we can aim for a larger dataset, but within a limited budget (these experiments already cost us more than $2,000 in invoking the APIs) and out of consideration for efficient and environmental-friendly experimentation, we strongly believe 100 is a reasonable number.

Coding attributes provided by GPT-3.5. We agree with the reviewer that the quality needs to be ensured. Towards this purpose, the several authors on this paper have (independently) manually validated the quality of the responses given by GPT-3.5–-detailed procedures and results can be found in Appendix E.3 and Table 3, referenced at line 313 in the main paper. The scores in Table 3 show that the extracted results given by GPT-3.5 are highly accurate and reliable. In addition to the attributes extracted by GPT-3.5, we also included attributes obtained from other means to serve as a supplement wherever possible, e.g., for measuring “efficiency”, in addition to the “complexity” results provided by GPT-3.5, we also measured the runtime and memory usage as a support. We hope these together increase the credibility of the results.

Humans prefer positive reviews. We are happy to provide some references [1,2,3]. We additionally want to point out that, what we mean here is not merely human preference in general, but human preference when specifically infused in preference tuning (like RLHF). It’s probably more accurate to put it another way: “preference tuning enforces the preference for positivity, correctness, and efficiency for a language model assistant.“

References

[1] Dodds, Peter Sheridan, et al. "Human language reveals a universal positivity bias." Proceedings of the national academy of sciences 112.8 (2015): 2389-2394.

[2] Augustine, Adam A., Matthias R. Mehl, and Randy J. Larsen. "A positivity bias in written and spoken English and its moderation by personality and gender." Social Psychological and Personality Science 2.5 (2011): 508-515.

[3] Boucher, Jerry, and Charles E. Osgood. "The pollyanna hypothesis." Journal of verbal learning and verbal behavior 8.1 (1969): 1-8.

Thank you for your review. We offer more discussion with respect to the concerns raised.

First, we agree with the reviewer that human evaluation could further help increase the credibility of evaluations and we have in fact made efforts towards this. Several authors on this paper have (independently) manually validated the quality of the responses given by GPT-3.5–-detailed procedures and results can be found in Appendix E.3 and Table 3, referenced at line 313 in the main paper; this was provided as part of the original submission. The scores in Table 3 show that the extracted results given by GPT-3.5 are highly accurate and reliable.

More broadly, the reviewer raised a valid point regarding the discriminative capability of LLMs. The question itself is interesting and has not been well studied yet. But there is evidence supporting contrasting sides of the argument [1,2], with [1] claiming that LLMs are better discriminators while [2] criticizing the understanding of LLMs. But, whether LLMs can be good discriminators or not is orthogonal to the message of our work; the conclusions we derive with respect to the diversity of the generations shall not hinge on the answer to this question. This is because 1) we have decomposed the discrimination task to a simpler form (e.g., sentiment classification, an easy task with human provided labels for ground truth exist for many datasets, and classifiers are trained for this very task) and 2) for these tasks, we have made sure to adopt well-trained models that are commonly used.

Thank you very much and please let us know if you have follow up questions or concerns.

References

[1] Are Language Models Better at Generating Answers or Validating Solutions? https://openreview.net/pdf/d3f724811cb971ff60cd45abfc51bd92d2dd7602.pdf

[2] West, Peter, et al. "THE GENERATIVE AI PARADOX:“What It Can Create, It May Not Understand”." The Twelfth International Conference on Learning Representations. 2023.

We thank the reviewer for their thoughtful comments.

Weaknesses

We briefly respond to the weaknesses mentioned by the reviewer. Overall, we agree that these are fascinating questions.

With regards to weakness 1+3, we would like to stress that our primary contribution in this work is defining generative monoculture, and providing a methodology to measure it. While changing temperature and prompting strategies were not effective countermeasures to deter generative monoculture, these are important negative results as these are methods that are often understood to promote diversity and showcasing their ineffectiveness is important. Our work shows that the alignment process might be one cause for the monoculture phenomenon; however, designing and evaluating solutions in the alignment phase for popular models are both non trivial and expensive. One possible approach would be to consider alignment approaches such as DPO [3] which introduce a flavor of supervision, or perform alignment using a “mixture of reward models” with diverse preferences. However, it is unclear what implications these solutions may have on downstream effects, such as output saliency, coherence, correctness and safety. Exploring this will be interesting future research.

For weakness 2, we agree that there are several cases where monoculture can be extended, such as studying how political preferences between train and model-generated data change. However, we believe that the two case studies we have picked are already representative. Extending monoculture evaluation to chatbots is unclear: what would the source distribution be? Measuring diversity change in this scenario would also require the nature and turn of dialogues to be the same as in this source distribution (for a fair comparison); this makes the scenario extremely convolved and hard to study.

For weakness 4, we will certainly study the literature associated with bandits and reinforcement learning, and make appropriate connections in our work. Thank you for the pointers!

Q1 and Q2. Feedback Loops/Model Collapse and Monoculture: a) Overview and comparison: First, we quickly remind the reviewer that the phenomenon that we demonstrate, generative monoculture, is separate from that of model collapse. Generative monoculture demonstrates that LLMs trained entirely on human-generated data reduce diversity/do not reflect the true range of information in their output. Model collapse shows how LLMs trained on their own generated data degrade in performance over time [1,2]. One can think of monoculture as a special case of collapse, where degradation is akin to progressive output diversity reduction.

b) (Q2) Strategies preventing model collapse, and its relation to our work: Research on the (separate) phenomenon of model collapse has suggested that access to the original data is crucial [2], and noted that random sampling could serve to mitigate the issue [1]. Importantly, there are different conclusions about these two related, but separate phenomena. We show that: Even LLMs trained on human (not model-generated) data have a narrower distribution than their training data. Despite the comments in [2] regarding the importance of original data–here, even with access to the original data, the issue of losing information in the true distribution still exists. Tuning sampling parameters cannot effectively mitigate the issue, contrary to what is suggested in work related to collapse [1].

Together, these suggest that in generative monoculture, the loss of diversity from training data to model responses is complicated. Results (and findings) from work on feedback loops and model collapse may not directly transfer here, and not all solutions/mitigations for model collapse may not work effectively in terms of mitigating generative monoculture. As stated earlier, we regard generative monoculture as a close but distinct topic to feedback loop / model collapse that deserves separate attention (we have also involved some other discussions at the end of Sec 7).

Regarding mitigating generative monoculture, we look forward to this in future work, but reiterate some preliminary thoughts here. Our analysis pointed out that generative monoculture is more of a concern for fine-tuned models (both instruction fine-tuned and RLHF fine-tuned, and more pronounced on the latter) and less of an issue for pre-trained models. Thus we would suggest making improvements to the fine-tuning data or the preference tuning approach to explicitly encouraging favoring diversity. One way to realize this is, instead of strictly optimizing the score of a single reward model, we optimize against a mixture of reward models with different preferences (look at response to weakness 1+3). Thus the diverse answers all have their chance to be promoted, instead of only one answer.

We thank the reviewer for the comments and questions and would like to offer some clarifications and hope that addresses some of the reviewer’s concerns.

Incomplete submission (appendix). Our appendix can be found in the supplementary materials. This was submitted along with the original paper, in accordance with the guidelines specified where the appendix could be part of the main paper, or be included with the supplementary material. We hope that since this was the main reason the reviewer cited for giving us a score of 1, the reviewer will raise their score.

LLMs for attribute extraction. The reviewer made a good point in that we should use care in attribute extraction. We hold the same belief that the extraction functions need to be “efficient, accurate, and reproducible” (line 161). Towards this purpose, the several authors on this paper have (independently) manually validated the quality of the responses given by GPT-3.5, not only for the “efficiency” attribute but also other attributes like “description” and more–-detailed procedures and results can be found in Appendix E.3 and Table 3, referenced at line 313 in main paper. The scores in Table 3 show that the extracted results given by GPT-3.5 are indeed highly accurate and reliable. In addition to extracted “efficiency”, we also measured the actual runtime and memory usage of the generated code (i.e., using system profiling tools), to complement the evaluation on “efficiency”. We hope these together increase the credibility of the results.

Human preference. We want to clarify that what we meant was not human preference in general, but the human preference specifically infused in preference tuning (like RLHF). It’s probably more accurate to put it another way: “preference tuning enforces the preference for positivity, correctness, and efficiency for a language model assistant.” We are happy to provide citations for the idea that humans prefer positive text [1,2,3]: the seminal work of Dodds et al. [1] states that “Using human evaluation of 100,000 words spread across 24 corpora in 10 languages diverse in origin and culture, we present evidence of a deep imprint of human sociality in language, observing that (i) the words of natural human language possess a universal positivity bias, (ii) the estimated emotional content of words is consistent between languages under translation, and (iii) this positivity bias is strongly independent of frequency of word use.” Similarly, the work of Augustine et al. [2] states that “The human tendency to use positive words ("adorable") more often than negative words ("dreadful") is called the linguistic positivity bias. We find evidence for this bias in two studies of word use, one based on written corpora and another based on naturalistic speech samples.“

Nitpicks. Those are all great suggestions for improving the clarity and readability of the paper, the presentation of the figures and formatting of the references. We will incorporate them in our revision.

Topics. For the clarification on topic models: each topic id corresponds to a topic that’s characterized by a list of keywords (we showed 2); together, these keywords are most representative of the topic and can be thought of as a concrete way to understand the embedding of the topic. Assignment of topic is not done by matching with these keywords but by comparing the text embeddings. So it’s normal that “novel” can occur in multiple topics, as these topics could concern just different genres of novels.

{person}. The list can be found in Appendix B.2. The list of names are as follows: Trevor Noah, Janelle Monáe, Yuval Noah Harari, Serena Williams, Reshma Saujani, Neil deGrasse Tyson, Margaret Atwood, David Attenborough, Malala Yousafzai, Jordan Peele.

Generating code solutions. To clarify our process: we kept generating solutions and measuring their correctness, until at some point all problems reached at least 20 correct solutions; we stopped then. That is why we ended up generating 100 solutions per problem for GPT-4 and 200 for Claude-3–essentially, GPT-4 is better at generating correct solutions and we stopped early for it while we had to continue generating more for Claude-3. Comparing Fig. 5a with Fig. 26 in Appendix, we can see that GPT-4 does achieve higher accuracy in general compared to Claude-3.

Thank you for your detailed response.

I would not be opposed to this paper getting accepted, but I do think it could benefit from another round of revisions.

Comments on Dataset Size + Cost We also note some reviewers had comments on the size of our coding dataset— we clarify that regarding the set of 100 problems, this means we generated and autojudged 10,000 solutions (100 x 100) in total by GPT-4 (and 20,000 solutions in total by Claude-3) on these 100 problems (described in “Generation and correctness evaluation of code solutions”). Moreover, the results on 100 problems already revealed pretty significant trends. It is true that we can aim for a larger dataset, but within a limited budget (these experiments already cost us more than $2,000 in invoking the APIs) and out of consideration for efficient and environmental-friendly experimentation, we strongly believe 100 is a reasonable number.

Additional Clarifying Edits to the Paper We have gone ahead and updated several wording changes in the paper (e.g. human generated→ human-written) and how exactly we express our Takeaway #3 in response to reviewer comments. We appreciate the reviewers' help to improve our presentation.

We have also clarified common misunderstandings about aspects of our experimental design, such as how and why we only show the diversity of the distribution of correct code outputs from the LLM. However, we note that several comments about clarity and incompleteness were noted by a reviewer who claimed we did not have an appendix when we did.

We hope that our clarifications, additional experiments, and edits answer all of the questions the reviewers had about our approach and results—and ask reviewers to clarify if they have not. We welcome follow-ups and further questions for the remainder of the discussion period.

[1] Are Language Models Better at Generating Answers or Validating Solutions? https://openreview.net/pdf/d3f724811cb971ff60cd45abfc51bd92d2dd7602.pdf