Does Writing with Language Models Reduce Content Diversity?

(2/2)

Addressing the concern that the reduction in diversity is attributed to the text contributed by InstructGPT is obvious: While the high-level finding might align with the view of researchers, in the collaborative setting it is not necessarily clear that the model-generated text is the only possible reason for these effects as recent work has also shown that interacting with the models might result in the users altering their behavior [1]. As a result, we compare the key points introduced by the humans and models in both the GPT3 and InstructGPT setups and find that the human contribution in both setups is almost identical, while the model-attributed key points cause the difference in diversity.

[1] Maurice Jakesch, Advait Bhat, Daniel Buschek, Lior Zalmanson, and Mor Naaman. 2023. Co-Writing with Opinionated Language Models Affects Users’ Views. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI '23). Association for Computing Machinery, New York, NY, USA, Article 111, 1–15. https://doi.org/10.1145/3544548.3581196

To answer the questions: Q1) Were the users instructed to read and revise their essays? Users are allowed to edit their essays post-completion. Their objective is to convey their opinions effectively. We acknowledge that this might affect word usage patterns which again motivates the contribution of our proposed key point-based evaluation as these key points are less likely to be changed during post-editing.

Q2) Are the recruited writers native English speakers? The users in this study are indeed largely native English speakers from the United States recruited from Upwork.

Q3 and 4) Why was introducing model-generated text not enforced for the group writing with model help? And relatedly, what percentage of essays were written by humans? We don’t enforce that users include a certain amount of machine-generated text as we want to study how collaborative writing in its most natural form differs from solo writing without model help. The average machine-written fraction is 0.32 for InstructGPT and 0.35 for GPT3 (Table 1) indicating that users obtain useful help from the models without over-relying on the same. The vast majority of essays have machine-written fractions ranging from 0.2 to 0.5 with a few outliers being as low as 0.1 and as high as 0.8 (Figure 12 in Appendix C).

Q5) Can we quantify the fraction of the tokenizer used in the generated text? What does prior work discuss about the reduction in diversity due to instruction tuning? The usage of the overall vocabulary i.e. the fraction of words from the tokenizer that are included in the generated text could be an interesting signal of creativity, however, the measure we would be really interested in is actually the fraction of words that are also relevant to the topic being written about. This is potentially more challenging to calculate. We do note that the data we collect, including all generated model suggestions is released publicly enabling further analysis along these lines. The specific findings discussed in prior work about the reduction in diversity include the reduction in the entropy of the output distribution after training with RLHF (Figure 15 in [2]) and the observation that instruction tuning reduces the calibration of the GPT-4 (Figure 8 in [1]).

[1] OpenAI. 2023. Gpt-4 technical report.

[2] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862

(1/2) Thank you for taking the time to review our paper and providing valuable feedback and questions. We appreciate your comments on the timeliness and presentation of our paper and the value of collecting and releasing the dataset of writing logs + essays to facilitate research in the area.

To address the concern about the comprehensiveness of reporting metrics studying the linguistic aspects of the collected essays: We used metrics based on n-grams and key points to measure content diversity in an interpretable way. We acknowledge that including syntactic and information-theoretic metrics would lead to a more holistic analysis of the collected essays and hence we will include the following summary statistics. We add these to Table 9 in Appendix C. We would like to highlight that releasing the essays collected would allow further in-depth analysis of the same.

1. Predictably, the essays written with model help have a lower average perplexity (or entropy) as measured via GPT2. This effect is more pronounced with InstructGPT than GPT3, complementing the results on n-gram and key point diversity, again highlighting that writing with model help can result in more generic content.
2. We find that the essays from each group are roughly the same length. When writing with InstructGPT, we find that users write slightly shorter sentences. Additionally, by calculating the average height of the dependency parse trees of the sentences in the essays, we observe that writing with InstructGPT results in sentences with fewer nested structures, indicative of slightly lower complexity.
3. Writing with model help also results in fewer unique POS-Ngrams again providing some evidence that collaborative writing could result in more homogenized usage of language. The effect here is even between GPT3 and InstructGPT, a slight departure from the results on diversity which is more focused on content. We finally plot the distributions of the 50 most frequent POS-Ngrams and find that on 4 and 5-grams, writing with model help leads to higher repetition of frequent POS-Ngrams.

Results in markdown (Same as Table 9 in the paper, numbers in brackets indicate standard errors on the computed average values):

Regarding the differences to CoAuthor:

1. While CoAuthor presented an observational study on the users’ writing processes with model help, we aim to establish the relationship between the intervention of model help and the diversity of content produced. As a result, we collect not only collaborative essays using different models but also essays written by the same authors without model help (which is missing in CoAuthor). CoAuthor primarily analyzed the impact of the sampling temperature on various metrics such as vocabulary and machine-contribution.
2. We also introduce the key point-based metric to better capture the content diversity between the setups. We believe that as research on the societal implications of introducing models in public-facing settings progresses, we would need to go beyond just traditional metrics in the literature measuring grammaticality and perplexity and our work provides a new axis of LLM evaluation in these real-world settings.

(2/2)

We compare the homogenization of essays generated by GPT3 and InstructGPT without any human intervention in the Figure 14 in the updated draft. We prompt the models to collect the essays with 10 variants of the following sentence ‘Write a three to four-paragraph argumentative essay on this prompt in the first person: + {Essay Prompt}’. We collect 10 essays on 5 topics, sampled with a high temperature of 0.9, and report the homogenization scores below. Overall the trend shows that essays generated with InstructGPT show higher homogenization across all topics, with both models showcasing less variance within each setup than the collaborative experiments with human users. While not directly comparable to the Solo or collaborative essays due to the different way in which the data was collected, these results further motivate our analysis from Section 6 that the contribution from the models is impacting the diversity and not a change in human behavior in the presence of the model.

We’ve updated Figure 4a per your feedback.

Q1: If we use GPT-3.5 to summarize the essays for key points, will the model-generated text be more similar than the human-written text. While the key points are generated by GPT 3.5, this effect is even across all setups i.e. we use the same GPT-3.5 model to summarize the essays even from the Solo-writing format. As the same model is used to summarize key points from each setup, the effect is averaged out and does not impact the results themselves.

(1/2)

Thank you for taking the time to review our paper and providing valuable feedback and questions. We appreciate your comments on the value of connecting LLM evals in research to real-world scenarios as well as using LLMs to create higher-quality metrics for measuring complex effects such as content diversity.

Regarding the surprisingness of the results: While InstructGPT has been shown to produce less diverse text than GPT3 (corroborated by our observations in Figure 12 and Section 6), in the collaborative setting human intervention (e.g., they can edit or reject model suggestions) may mitigate this impact of the model. Indeed despite only 32% of the essays contributed by InstructGPT and 35% by GPT3, or around 66% of the essays written by the human users, we observe that the contribution of InstructGPT is sufficient to reduce the overall diversity of the generated essays. While this might agree with the intuitions of researchers, we believe the value of our findings lies in quantifying and measuring this effect rigorously, given the unclear effect of human interaction in the collaborative process. By making the collected data public, we hope to foster further research along these directions.

To address the concern about the generalizability of our findings to other LLM-based writing tools. While our results focus on a set of OpenAI models, we believe they hold significance due to two key reasons. Firstly, many tools released by various companies are built upon OpenAI APIs as they are recognized for their high performance (https://crfm.stanford.edu/helm/latest/?group=core_scenarios) and usability in products. Secondly, our chosen models enable a comparison between a general-purpose base LLM (davinci) and an RLHF-trained equivalent (text-davinci-003), allowing us to generalize the distinction in training methods to other model families. Additionally, in our collaborative writing setup, the continuations are sampled from the model given the current user draft as the prompt i.e. we can test the model impact averaging over many different prompts, again contributing to the generalizability of these results. Finally, to address the aforementioned findings from the WordCraft project that LLMs can help dislodge writers from their common patterns, we believe that our results complement their findings and highlight a potential concern as well. Consider the scenario where a professional author is stuck while writing and accepts model suggestions to dislodge themselves from their block. While that writer’s productivity improves, it becomes a concern if similar ideas to shift out of a block are used by many different users. Our findings suggest that this improvement in individual productivity could potentially arise at the cost of a homogenizing effect, resulting in less colorful stories in the long run. In the WordCraft paper itself, it was noted that the model often resorted to standard tropes as suggestions. This motivates why we chose to study the effect of model intervention in aggregate across different users and introduce quantitative metrics to observe the reduction in diversity.

Clarifying how the fraction of model intervention is calculated: We clarify that if a model generates a sentence and the user fully rewrites that sentence, then the machine-written fraction here would be zero. Our text editor interface tracks if each character was most recently introduced by the model generation or by the user’s keystrokes by recording each change of character in the draft. This is what we use to compute the model contribution and attribute key points to the model and user which we use for analysis in Section 6. While this does not include the contribution of the user merely seeing the model suggestions and then paraphrasing these into the draft, we do record every single suggestion shown to the user and this can be matched up to the final artifact produced by the user. Hence we again believe that our interface and collected data could serve as a valuable resource to the community.

Thanks a lot for your extensive review. As the discussion period is nearing its end, please let us know if you have any remaining questions. We hope that we were able to convey the novelty of our findings and clarify your other questions. We will continue to update future versions of the paper based on your feedback!

(2/2)

W4: To address the concern about simple confounding factors to the observed diversity effects that could be addressed by reporting more statistics about the collected essays: We aim to improve the comprehensiveness of our analysis by reporting additional metrics on the collected essays as follows. We add these to Table 9 in Appendix C.:

1. We computed basic statistics such as perplexity, average sentence length, and essay length on the essays collected. Predictably, writing with model-help reduces the perplexity of the essays which now incorporate suggestions sampled from an LM distribution. The total length of essays, in terms of word count, is roughly similar, with writers writing slightly shorter sentences when writing with InstructGPT (albeit with a slightly high standard error).
2. To ensure that the quality of the essays is not a confounder in the result, we present human ratings for a random sample of 10 essays (one per topic) from each setup that we conducted as part of our quality control checks. Here we judged the essays on 3 different axes - relevance to the topic, grammaticality, and the depth of discussion on their opinions. This was per the advice from faculty at the Expository Writing program at our university. This analysis showed no clear differences between the setups, explained in part due to the observation that the machine-written fraction in the collaborative essays was on average around 32%-35% (Table 1).

Results in markdown (numbers in brackets refer to the standard errors on calculating the average values in the cells):

Q2: Clarification that the model sizes of the chosen base LLM (davinci) and RLHF model (text-davinci-003) are comparable: The OpenAI model index (https://platform.openai.com/docs/model-index-for-researchers) confirms that the davinci endpoint was the 175B parameter model released with the original GPT3 paper i.e. a pure LLM, and that text-davinci-003 was of the same size with subsequent training with RLHF (PPO).

(1/2)

Thank you for taking the time to review our paper and providing valuable feedback and questions. We appreciate your comments on the importance of defining and quantifying metrics for measuring the effect of model intervention on content diversity. We discuss some of your concerns below

W1: To clarify the experimental setup and address the concern that an independent samples t-test might not be appropriate for the chosen experiment design: We recruited a fixed set of users who wrote essays in each of the three setups in a randomized order. We ensure that they write essays on different topics in each setup to prevent a repetition of their opinions. However, this doesn’t fall clearly into the within-subjects study as the topic which they write on is different, thus we cannot create pairs of essays written by the same authors in different setups since these will be on different topics (which is a confounder). As a result, we performed significance testing via independent t-tests.

To address your concerns regarding statistical significance, we observe that within each topic, each writer only submits an essay to one of the three setups. This matches the between-group setup, albeit with less power as we only collected 10 essays per setup per topic. We compute the significance of the difference in homogenization scores per topic via an independent samples t-test. The p-values are reported in tables 7 and 8 of the draft comparing each pair of setups within each topic at both the key point and essay level. Comparisons with significance at the 5% level are marked with an asterisk. We observe that the InstructGPT setup exhibits higher homogenization over Solo writers with significance at the 5% level on 8 out of 10 topics at the raw essay level and 6 out of 10 at the key point level using both Rouge-L and BertScore. The same trend holds on GPT3 on 7 topics at the raw essay level and 5 at the key point level. We believe that this confirms the overall trend that writing with InstructGPT results in higher homogenization in the essays created and appreciate your note to improve the rigor of our testing. We have added this to the paper in Appendix B.3, tables 7 and 8 including a forward reference to these results in Section 4.2.

W2: Addressing the distinction between concepts of homogeneity and diversity being measured in the paper: In general, homogenization measures whether the outputs from a person are similar to the outputs of another (or group of people) while diversity measures how much redundancy there are in a set of outputs from a group of people. As noted, these concerns are related whereby high homogeneity (i.e. documents are quite similar to one another) between essays would correlate with lower diversity (i.e. key points made across documents are repeated). From a practitioner’s perspective, if you are interested in the authenticity of the text, then you would want to measure homogeneity which indicates whether the author’s voice is lost during co-writing. If you want to study the aggregate property of text, diversity measures how language as a whole changes when the writing process is intervened by LLMs.

W3: Discussing how variations in prompting strategies would impact the results of our study: While we do not investigate the role of the formatting of prompts, in this work since we sample continuations from the models for the current draft, the essays as written by different users serve as the prompts to the model for each query. However, the homogenization and diversity effects we observe occur despite this variance in prompting. A deeper exploration into how humans would prompt models to potentially mitigate this issue is a direction of future research, although it is unclear how different the prompts generated by different users would be as recent findings have shown that users do not systematically explore methods of prompting [1].

[1] J.D. Zamfirescu-Pereira, Richmond Y. Wong, Bjoern Hartmann, and Qian Yang. 2023. Why Johnny Can’t Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI '23). Association for Computing Machinery, New York, NY, USA, Article 437, 1–21. https://doi.org/10.1145/3544548.3581388

Thanks a lot for your extensive review. As the discussion period is nearing its end, please let us know if you have any remaining questions. We hope that the additional results analyzing the essays and statistical tests presented convey the significance of the results and will continue to update future versions of the paper based on your feedback!

I thank the authors for their reply to my W1. Now I understand why the authors use independent-sample t-tests despite that the same group of subjects were used in the three different tasks. I agree with the authors that due to the differences in the topics, this is better treated as a between-subject design.

Regarding W2, while the authors' reply helped a little, I still feel the two concepts ("homogeneity" and "diversity") will be confusing to readers. I trust the authors will make an effort to clarify this as much as possible in revisions.

As for W3, thanks for acknowledging that users' own instruction and prompting during LLM-assisted text composing is a direction for future research (and should be mentioned in the revised paper). I agree that not directly addressing instruction and prompting in the current paper is fine.

As for W4, the table looks informative and comprehensive. I appreciate the authors effort in performing the additional analysis.

As such, I'm willing to updating my rating from "5: marginally below the acceptance threshold" to "6: marginally above the acceptance threshold" if that's allowed by the ICLR review process.