AI as Humanity’s Salieri: Quantifying Linguistic Creativity of Language Models via Systematic Attribution of Machine Text against Web Text

On another note authors claim about DetectGPT being the strongest zeroshot AI text detecting system is not true. Please refer to this paper (https://liamdugan.com/files/raid.pdf)

Binoculars [https://arxiv.org/abs/2401.12070] is a lot better than DetectGPT. Actually DetectGPT is a lot worse than several content detectors.

In the light of this and no rebuttal from authors inspite of several concerns raised I am leaning towards reducing my scores. Authors if you think there are issues with the points raised in my review please respond as the discussion period is ending

Thank you for bringing up more recent work in zero-shot machine text detection! We have revised the paper to remove the claim that DetectGPT is the strongest zero-shot detector. In the next revision, we will include comparisons with more recent detection systems, such as Binoculars. The primary goal of this paper is to analyze statistical differences between professional human writing and seemingly amazing outputs from LLMs. Applying the CREATIVITY INDEX to machine text detection is intended as an exploratory effort for novel applications.

We also apologize for the delayed response. The additional experiments took a while to conclude due to limited resources. Additionally, the extension period has been extended to December 2nd!

We thank the reviewer for their detailed feedback, constructive critique, and recognition of the importance of our work in exploring how “trace text from pre-training is an excellent opportunity for the future of work and creative labor.” We also appreciate the acknowledgment of our “detailed experiments,” the “intuitive” nature of our idea, the “great structure and organization of the paper,” and the “very valuable contribution from an engineering standpoint.” Below, we address the key concerns and questions raised.

Re: Ablation for Decoding Temperature

Thank you for the suggestion to experiment with decoding temperature! We fixed p=1.0, varied the decoding temperature from 0 to 1, and computed the CREATIVITY INDEX. We observed that a higher temperature resulted in a marginally higher CREATIVITY INDEX. Specifically, ChatGPT's CREATIVITY INDEX was 8.4% higher with a decoding temperature of 1.0 compared to 0 (p = 0.16; N = 600). However, this difference was minimal compared to the gap between humans and LLMs. The CREATIVITY INDEX of human authors was 49.1% higher than that of ChatGPT at a decoding temperature of 1.0 (p = 6.9 × 10^{-17}; N = 600).

Re: Whether Proposed CREATIVITY INDEX Measure Creativity

Thank you for the suggestion to expand on this discussion! We respond below and have also updated the paper draft to to include additional discussion.

As we mentioned in the paper, CREATIVITY INDEX is designed to measure one specific aspect of creativity—linguistic creativity (i.e., the novelty in composing words and phrases). This is approximated by computing originality of the text with respect to a large pre-training corpus. While effective at capturing statistical differences between professional human writing and seemingly remarkable outputs from LLMs, it may not comprehensively reflect other dimensions of creativity, such as flexibility or elaboration as featured in the Torrance Test. Therefore, the CREATIVITY INDEX alone is insufficient for drawing definitive conclusions about overall creativity, especially when analyzing nuanced variations among human authors.

Additional factors, such as writing style and the time of composition, can also influence the CREATIVITY INDEX. For example, older English writings may exhibit higher scores, as they are more difficult to reconstruct from the pretraining corpora of modern English. Furthermore, CREATIVITY INDEX assumes the input text is of sufficient quality, as our study focuses on outputs from recent LLMs that are already fluent and coherent. When applied to broader contexts, the CREATIVITY INDEX can be complemented by standard automatic measures of text quality, such as fluency classifiers or perplexity-based evaluations, to provide a more comprehensive assessment. We have incorporated discussions of these considerations into the paper.

To further validate the CREATIVITY INDEX, we computed its correlation with existing human evaluation-based creativity metrics, specifically the Torrance Test of Creative Writing (TTCW) proposed by Chakrabarty et al. [1] This framework evaluates the creativity of stories written by LLMs and human authors using expert evaluations based on a carefully designed rubric inspired by the Torrance Test of Creative Thinking (TTCT) (See General Response).

We found that the CREATIVITY INDEX positively correlates with all dimensions of TTCW, showing particularly strong correlations with high statistical significance in dimensions such as “Emotional Flexibility,” “Narrative Pacing,” “Structural Flexibility,” and “Originality in Thought.” Again, as the CREATIVITY INDEX is specifically designed to measure linguistic creativity, it may correlate with other dimensions of creativity but does not comprehensively capture all of them. Therefore, it serves as a complementary metric to existing creativity measurement methods.

[1] Chakrabarty, Tuhin et al. “Art or Artifice? Large Language Models and the False Promise of Creativity.” Proceedings of the CHI Conference on Human Factors in Computing Systems (2023): n. pag.

(1 / 3, to be continued)

We thank the reviewer for their encouraging and constructive feedback! We are delighted to hear that they consider our work to have “significant practical impact,” and we greatly appreciate their recognition of its “solid theoretical foundations, practical implementation details, and clear engineering considerations.”

CREATIVITY INDEX on Multilingual Data

Thank you for the suggestion to expand our analysis to multilingual data! We conducted additional experiments to compute the CREATIVITY INDEX for ChatGPT and human authors in French and German within the novel domain.

For human-written French novels, we randomly sampled 10 books from the “Contemporary French Books” list on Goodreads (https://www.goodreads.com/shelf/show/contemporary-french). From each book, we randomly selected 10 snippets (i.e. continuous sequences of K sentences with a total length exceeding 256 words) as the human text. Similarly, for German novels, we randomly sampled 10 books from the “Modern German Novel Books” list on Goodreads (https://www.goodreads.com/shelf/show/modern-german-novel). For machine texts, we prompted ChatGPT to generate continuations for these novels starting with an initial sentence from the human writings, following the same setup as in the paper.

As a reference corpus, we used the Common Corpus from PleIAs, the most recent multilingual pre-training dataset. Due to time and resource constraints, we were not able to build an Infini-gram search index for the entire corpus. Instead, we subsampled approximately 20B tokens each from the French and German portions of the Common Corpus to serve as the reference.

We found that human writings consistently exhibit a significantly higher CREATIVITY INDEX than ChatGPT in both French and German, aligning with our main results. Specifically, the CREATIVITY INDEX of human authors was 46.1% higher than ChatGPT in French novel writing (p = 7.5 * 10^{-13}; n = 100) and 54.3% higher in German (p = 1.2 * 10^{-15}; n = 100). We plan to expand these experiments to additional languages in the next revision.

Additional Ablation for Deduplication Threshold

Thank you for the suggestion to expand our ablation on deduplication thresholds! We analyzed the relationship between the degree of overlap S(x,d) and the likelihood that the reference document d contains an exact copy or quotations of the input text x.

We found that when S(x, d) < 0.3, there are no exact copies, and almost no quotations of the input text x in the reference document d. Conversely, when S(x,d) > 0.9, the reference document d almost always contains an exact copy of x. For cases where 0.3 < S(x, d) < 0.9, all three scenarios (exact copy, quotation, or none) are plausible. Therefore, we chose to prompt an LLM to determine whether d contains exact copies or quotations of x in this range, ensuring accurate deduplication.

Maintain and Update Reference Corpus

The effectiveness of our method indeed relies on the quality and coverage of the reference corpus. Building a high-quality and high-coverage pre-training corpus remains an important research question that the community is actively addressing with notable progress. At the time of our experiments, RedPajama was the largest open-source pre-training corpus available. Since then, newer large-scale pre-training corpora, such as DOLMA [1], have been released and incorporated into the infini-gram search API. Additionally, Common Corpus from PleIAs, which includes extensive multilingual data, is another resource we plan to integrate into the infini-gram search API.

We will continue to monitor advancements and integrate new pretraining corpora into DJ SEARCH as part of the reference corpus. Additionally, one of our future directions is to create our own large-scale, reliable reference corpus with fast search capability.

Open-Source DJ SEARCH

Yes! We will definitely open-source the code, and we also provide an online demo of DJ SEARCH for easy and accessible use.

[1] Soldaini, Luca et al. “Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research.” ArXiv abs/2402.00159 (2024): n. Pag.

[2] PleIAs. “PleIAs/Common_corpus · Datasets at Hugging Face.” PleIAs/common_corpus · Datasets at Hugging Face. Accessed November 26, 2024. https://huggingface.co/datasets/PleIAs/common_corpus.

We thank the reviewer for their constructive comments and appreciate the acknowledgment of the paper as “well-crafted,” “greatly aligned with the intention,” and conducting a “multifaceted and multiangled extension of discussions.”

Re: Timeliness of Data

As mentioned in the original draft, “the computation of the CREATIVITY INDEX is constrained by the reference corpus used for DJ SEARCH.” We acknowledged that “Without incorporating private data used to train closed source LLMs such as ChatGPT into the reference corpus of DJ SEARCH, the CREATIVITY INDEX of closed-source LLMs may be somewhat inflated.” Similarly, we agree that human writing from older times might also exhibit somewhat inflated CREATIVITY INDEX, as the reference corpus may not cover older English as comprehensively as modern English. We have added further discussion on this topic in the paper.

The main focus of this work is to understand the differences between seemingly amazing machine text and professional human writing by quantifying some aspect of linguistic creativity through the lens of n-gram comparisons with web text. Precisely distinguishing creativity differences within human writing remains a challenging research problem [1, 2, 3], which is beyond the primary scope of this work.

Nonetheless, it is interesting to explore whether the CREATIVITY INDEX remains robust when the reference corpus has much smaller coverage. To investigate, we conducted experiments using the older pre-training corpus, Pile, which is approximately 1/5 the size of RedPajama. Even with this much smaller reference corpus, we found that human writings still exhibit a significantly higher CREATIVITY INDEX than any LLM. Specifically, based on the Pile, the average CREATIVITY INDEX of humans was 42.8% higher than that of LLMs in novel writing (p=1.4 × 10^{-26}; n = 100), consistent with our main findings.

Re: Depth of Matching Method Exploration

In the original draft, we conducted detailed experiments to analyze the effect of semantic matching versus exact matching, as described in the experiment section “How do different matching criteria affect creativity measurement?”

We found that: “First, the creativity gap between humans and LLMs becomes even larger when considering semantic matches in addition to verbatim matches (Fig. 3d). Averaged across all models, the CREATIVITY INDEX of human, based on both verbatim and semantic matches, is 102.5% higher than LLMs in novel writing (p = 2.6 × 10−12; N = 600), whereas based on verbatim matches alone, the CREATIVITY INDEX of human is 52.2% higher than LLMs. Second, semantic matches provide more signal for analyzing the uniqueness of longer n-grams (Fig. 3e). For example, while the gap in L-uniqueness at L = 11 between human text and machine text from OLMo Instruct is 3.7% based on verbatim matches alone, this gap widens to 16.3% when considering both verbatim and semantic matches (p = 3.1 × 10−7 ; N = 600). This indicates that although some of the longer n-grams in machine text may appear unique at the verbatim level, they are similar to certain text snippets in the reference corpus at the content level.”

We thank the reviewers for their encouraging and constructive feedback! We appreciate the recognition of the “simple and easy to understand” idea, the “level of completeness in terms of depth of experiments, discussion coverage, and insight,” as well as the “quality visualization.”

Re: Runtime of DJ SEARCH

Thank you for the suggestion to include additional runtime analysis!

To verify linear complexity, we measured the runtime of DJ SEARCH with respect to increasing input text lengths using RedPajama as the reference corpus, averaging the results over 500 examples. The results are as follows:

We observed that the runtime of DJ SEARCH indeed increases linearly with the length of the input text.

We also measured the runtime of DJ SEARCH with respect to increasing sizes of the reference corpus while keeping the input length fixed at 256 words. In addition to RedPajama, which contains approximately 1.4 trillion tokens, we experimented with the Pile (380B tokens) and DOLMA (2.6 trillion tokens). The results are as follows:

We found that the runtime increase for DJ SEARCH is minimal as the size of the reference corpus grows. This is likely due to the high optimization of the Infini-gram, which enables it to retrieve matched n-grams within milliseconds.

Re: Compare with Previous Work

Thank you for the suggestion! We compared our CREATIVITY INDEX with the Torrance Test of Creative Writing (TTCW) proposed by Chakrabarty et al. Please refer to the General Response for further details.

We found that the CREATIVITY INDEX positively correlates with all dimensions of TTCW, showing particularly strong correlations with high statistical significance in dimensions such as “Emotional Flexibility,” “Narrative Pacing,” “Structural Flexibility,” and “Originality in Thought.”

Re: Variation of WMD

Thank you for the suggestion! We experimented with different variations of transport-inspired distances, including relaxed WMD, supervised WMD [1], and Sinkhorn distance [2].

We computed the CREATIVITY INDEX for ChatGPT and human writing in the novel domain based on semantic matches using relaxed WMD, supervised WMD, and Sinkhorn distance. Regardless of the distance metric used, human writings consistently exhibited a significantly higher CREATIVITY INDEX than ChatGPT, aligning with our main results. Specifically, based on relaxed WMD, the CREATIVITY INDEX of human authors was 97.6% higher than ChatGPT (p=2.5 × 10^{-13}; n=100). Based on supervised WMD, the CREATIVITY INDEX of human authors was 79.3% higher than ChatGPT (p=1.9 × 10^{-7}; n = 100). Based on Sinkhorn distance, the CREATIVITY INDEX of human authors was 91.3% higher than ChatGPT (p=3.5 × 10^{-12}; n = 100).

As mentioned in the original draft, the motivation for using transport-inspired distances—which combine word embedding distances for each n-gram's words rather than directly computing the Euclidean distance between embeddings of two n-grams—is rooted in efficiency. The latter requires computing n-gram embeddings for every n-gram in the input text and every document in the reference corpus during DJ SEARCH, which is computationally infeasible. Ideally, a good transport-inspired distance should approximate the Euclidean distance between the embeddings of two n-grams.

To evaluate this, we computed the Pearson correlation between our implementations of WMD, relaxed WMD, supervised WMD, and Sinkhorn distance with the Euclidean distance between the embeddings of two n-grams to assess how closely they correlate.

We found that all transport-inspired distances correlate highly with Euclidean distance, showing strong statistical significance. The slightly lower correlation with supervised MWD may be due to its training objective, which focuses on distinguishing between different categories of documents.

Re: Euclidean/Sinkhorn-based Distance

As mentioned in the original draft, the motivation for using transport-inspired distances—which combine word embedding distances for each n-gram's words rather than directly computing the Euclidean distance between embeddings of two n-grams—is rooted in efficiency. The latter requires computing n-gram embeddings for every n-gram in the input text and every document in the reference corpus during DJ SEARCH, which is computationally infeasible. We also experimented with Sinkhorn distance alongside various versions of Word Mover’s Distance; please see the response above.

Re: Reference Copus

At the time of our experiments, RedPajama was the largest open-source pre-training corpus available. Since then, newer large-scale pre-training corpora, such as DOLMA [1], have been released and incorporated into the infini-gram search API. Additionally, RefinedWeb and Common Corpus from PleIAs [3] have also become available, and we plan to integrate them into the infini-gram search API.

Building a high-quality and high-coverage pre-training corpus remains an important research question that the community is actively addressing with notable progress. We will continue to monitor advancements and integrate new pretraining corpora into DJ SEARCH as part of the reference corpus. Additionally, one of our future directions is to create our own large-scale, reliable reference corpus with fast search capability.

[1] Huang, Gao et al. “Supervised Word Mover's Distance.” Neural Information Processing Systems (2016).

[2] Cuturi, Marco. “Sinkhorn Distances: Lightspeed Computation of Optimal Transport.” Neural Information Processing Systems (2013).

[3] PleIAs. “PleIAs/Common_corpus · Datasets at Hugging Face.” PleIAs/common_corpus · Datasets at Hugging Face. Accessed November 26, 2024. https://huggingface.co/datasets/PleIAs/common_corpus.

Several reviewers suggested comparing our CREATIVITY INDEX with existing human evaluation-based creativity measures. A recent study by Chakrabarty et al. [1] evaluates the creativity of stories written by LLMs and professional human authors through expert evaluations based on a carefully designed rubric inspired by the Torrance Test of Creative Thinking (TTCT). This work introduces the Torrance Test of Creative Writing (TTCW), which consists of 14 binary questions grouped into four dimensions: Fluency, Flexibility, Originality, and Elaboration. To conduct the evaluation, ten expert creative writers were recruited to evaluate 48 stories—written either by LLMs or published in The New York Times,—using the TTCW rubric.

For each story, three expert evaluators are tasked with answering 14 binary questions, each designed to quantify a specific dimension of creativity. For example, within the Fluency dimension, "Understandability & Coherence" is assessed by asking, “Do the different elements of the story work together to form a unified, engaging, and satisfying whole?” In the Flexibility dimension, "Emotional Flexibility" is evaluated with the question, “Does the story achieve a good balance between interiority and exteriority, in a way that feels emotionally flexible?” Similarly, within the Originality dimension, "Originality in Thought" is measured by asking, “Is the story an original piece of writing without any clichés?” For each of the 14 binary questions, the final answer is determined by majority vote among the expert evaluators.

To evaluate the correlation between our CREATIVITY INDEX and Chakrabarty et al.'s TTCW, we compute the CREATIVITY INDEX for each of the 48 stories. For each binary question in the TTCT (e.g., "Understandability & Coherence" or "Emotional Flexibility"), we calculate the point biserial correlation between the CREATIVITY INDEX values and the corresponding binary expert ratings for that question across all 48 stories, as shown below.

We found that the CREATIVITY INDEX positively correlates with all dimensions of TTCW, showing particularly strong correlations with high statistical significance in dimensions such as “Emotional Flexibility,” “Narrative Pacing,” “Structural Flexibility,” and “Originality in Thought.”

Quantifying creativity precisely remains a challenging research question, as different works adopt varying definitions of creativity [2, 3, 4]. Even when using the same carefully designed rubric, such as in TTCW, expert annotations do not always agree with each other due to subjectivity. In TTCW, the Fleiss' 𝜅 agreement across expert annotations ranges from 0.27 to 0.66, indicating low to moderate agreement. The primary goal of our paper is to explore the differences between seemingly amazing machine text and professional human writing by quantifying some aspect of linguistic creativity through the lens of n-gram comparisons with web text. While our approach provides valuable insights, it alone might not be a definitive measure of creativity or capture all dimensions of creativity, and is therefore complementary to existing creativity measurement methods.

This is kind of a bogus critique in my opinion. In regards to "gaming" benchmarks, that's just par for the course for any kind of benchmark outside of LLM-as-judge or chatbot arena.

I think that there are many notions of "creativity" or "diversity", and nearly all of them will have pathological cases, including stuff worse than the case you outlined here.

More benchmarks, even with some issues, are better than less.

I will maintain my high score. This is great work.

Now coming to "Linguistic Creativity" . What is its definition ? Can you cite it ? What do you mean by it? From what I understand there needs to be certain degree of appropriateness. LLMs generate a hell lot of awkward phrases

Now here is why I think your metric has fundamental flaws. I took an excerpt from a George Saunders Flash Fiction that was published on August 29 2024 after Red Pajamas cut off data

Original : One Saturday morning, the man who had actually killed the former premier went to his dead uncle’s garage, where he’d hidden the red car, started it up, then sat behind the wheel, thinking. Once already, in the name of goodness, he’d sullied himself with murder. Shouldn’t he at least try to kill this new, even more evil, premier? Otherwise, what had it all been about?Killing the new premier would entail killing another set of unlucky diners. For him: more panic attacks, fresh nightmares. This time, if he succeeded, he would at least take some credit. Good God, yes, he would: no more silent suffering in anonymity. Afterward, he would climb up on the hood of his red car and, addressing the people, propose a new movement that would eschew violence and rule by justice and gentleness forever. Also firmness, of course. A merciful firmness the people would instantly respect.Or he could forget the whole thing, go back to being the man he’d been before the first killing: a frightened, passive nobody.This thought made him sicker than the thought of another round of screaming and crunching.He leaped out of the still running red car, threw the garage door open, and, feeling alive and decisive, stood blinking in the morning sunlight, readying himself for his task.

LLM Generated Paragraph with same content details : In the crimson Mazda's cocoon, he shudders as phantom screams pierce his consciousness—not his own, but those of the former premier, whose last breath had tasted of copper and surprise. His fingers dance across the dashboard, muscle memory recalling how easily the garrote had slipped around his target's throat, smooth as silk until it wasn't. The garage's stale air, thick with motor oil and his uncle's lingering cigarette ghost, feels like a confessional booth as he weighs his soul against the current premier's sins. Every sunrise brings fresh headlines of corruption that make his predecessor seem saintly, yet each night drags him through a carousel of terrors—dreams where his victims' faces morph into his own, where blood flows upward like reverse rain. The prospect of retreating into anonymous mediocrity nauseates him more than the memories; he's tasted the intoxicating wine of consequence now, sampled the heady bouquet of being someone who matters. His manifesto sits heavy in his breast pocket, each word a bullet waiting to reshape reality. The car door flies open with the force of destiny itself, and he emerges transformed—no longer a mere assassin, but a revolutionary being born in his dead uncle's garage, ready to paint his name across history in whatever shade of red fate demands.

Now both are written on the exact same content. I ran some quick experiments using Infinigram ( 5<=n<=11). The scores are Original George Saunders written Paragraph has a CREATIVITY INDEX of 59.5 while LLM generated paragraph has a CREATIVITY INDEX of 73.67 (I might be a little bit off with the numbers but these are close approximates)

Now I find it hilarious because George Saunders is one of the biggest writers alive (https://en.wikipedia.org/wiki/George_Saunders)

In the LLM generation these expressions are very very awkward (makes very little sense)

• garage's stale air, thick with motor oil and his uncle's lingering cigarette ghost, feels like a confessional booth (???)
• last breath had tasted of copper and surprise (???)
• each night drags him through a carousel of terrors (???)
• blood flows upward like reverse rain (???)
• tasted the intoxicating wine of consequence now, sampled the heady bouquet (???)
• each word a bullet waiting to reshape reality (???)
• ready to paint his name across history in whatever shade of red fate demands. (???)

I don't think these qualify as linguistically creative and now your metric has claim AI to be better than an expert

This is exactly what i feared because your metric will force people to call LLMs more creative because they produce fancy ngrams with no overlap in pre-training (even though this was not your intention). Memorization is interesting to study but when you use it to claim anything with real humans you have to understand the consequences.

How do you justify this ?