QUDsim: Quantifying Discourse Similarities in LLM-Generated Text

Thank you for your feedback! Addressing the concerns raised below:

technical details that may have an impact on the result are missing

We use the default parameters (temperature=1 and top-p=1). We included these details in Appendix B.

the generated texts depend heavily on the prompt and because prompts can be endlessly refined to produce exactly what you want

It is true that prompting can heavily impact the results, but we believe the question we are studying is natural for two reasons. First, relatively short prompts are often used in practice: for instance, the WritingPrompts dataset from reddit features short instructions, and human writers working on an initial draft may not have a large amount of content worked out.

Second, there is a fundamental question of how much of the space of human-written stories is being captured by LLMs. A short prompt represents an easy form of this task, as there are many possible answers, yet we see LLMs generating templatic ones. As a result, we see our findings as meaningful and still impactful even in the presence of longer prompts (perhaps until the prompt gets so detailed than LLM is just minorly revising an existing text, at which point we do not expect much deviation from the source).

human texts are longer than LLM-generated ones

LLMs have a “natural” length of responses they tend to generate, which may be long for simple queries but may fall short in longer-form generation settings, particularly something like an obituary which could cover a wide range of facts about a person. Furthermore, there would be differences in terms of words per sentence depending on the genre. Creative writing includes more dialogue which may be short phrases, while obituaries written by New York Times authors are more information-dense and descriptive.

how much would you expect generated-texts to approach the human-generated texts by refining the prompt?

We don’t see an easy way to precisely answer this question. We believe minor modifications of the prompt (“make it as human-like as possible” or variations of that form) will not help. In the limit, a prompt like “Here is a human-written obituary: [...] Please repeat it.” will of course match the human-written text. Providing a content plan matching the human one will give results somewhere in the middle.

could it be that there may be possible segmentations possible and that humans are just more creative in their interpretations?

Annotators were given unsegmented documents. So, it is possible that annotators may have intuitively grouped sentences together internally while reading for better comprehension - however, we did not ask them to perform segmentation. We saw that different annotators listed slightly different answer sentences, which may have been a result of this. However, we found that LLMs were more reliable at answering questions directly.

I'd appreciate some clarification on the statement regarding the exclusion of prior content to keep QUDs free of anaphoric expressions.

Since this work looks at cross-document answerability of QUDs, it is important for the QUDs to be fully specified and don’t have anaphors (compare: “What is the reason given for the deception?” to “What is the reason for that?”). We found that generating QUDs together with segment decontextualization (Appendices C3 and C4) is an effective way to achieve this.

Thank you for your feedback! Addressing the concerns raised below:

Proposed measure ignores the order of the textual segments… Other discourse theories like PDTB or RST could be more helpful here.

We agree that the lack of a hierarchical structure is a limitation of the current method, though for the ordering of segments, we have studied sequential QUD templates (Section 7). Theoretically, QUD structures can take the form of stacks (Roberts, 2012), trees (De Kuthy et al., 2018), or dependency structures (Wu et al., 2023, Ko et al., 2023). Yet, as a first step towards quantifying similarity with QUDs, we chose a structure-free (i.e., “flat”) option with a level of abstraction that can be operationalized (lines 80-85). The reason for this is to avoid theoretical complexity that can quickly get out of hand: the structures themselves assume some level of entailment relationship between the questions (in the case of stacks and trees), which is beyond the scope of this paper. (Note that this work already engages QUD with other semantic/pragmatic theories, such as alternative semantics in questions.) Similarities of dependency structures can apply to this work; however, in this paradigm, the QUDs are generated independently from each segment (also see Wu et al., 2023), thus the structure has no impact on the QUDs themselves but is solely about which sentences invoke the QUD at hand. Since QUDsim measures cross-document QUD answers, we believe that analyzing the QUDs themselves is more meaningful, and leave the structure to future work.

Similarly, it would be great to also use aspects of PDTB and RST, but these too are something we think would be exciting to investigate in the future: the intersection between QUD and coherence relation is a fascinating topic, but not a straightforward one (e.g., see Riester et al., 2021, Shahmohammadi et al., 2023, Ko et al., 2023, among many others). We intentionally chose the QUD theory because of its nature which bridges the content of discourse with the structure of discourse (for instance, see Ko et al., 2023 section 3, where QUDs can be used to characterize coherence relations), which we believe is a solid middle ground that does not drift into similarity strictly in terms of coherence structure. It is indeed very true that QUD is new to the type of investigation here, but this is exactly why we did it!

We will include more discussion around this point in any future version.

The dataset is very small for such a fine-grained analysis. Only 10 documents written by human are not enough for the established conclusions.

The focus of this work is a similarity measure, for which we have 100 documents in total. Most of these documents are from different LLMs, for us to draw conclusions about templaticness across LLMs. Our goal is not to study human-LLM divergences or establish the higher creativity of humans, which has been documented in other work (Padmakumar and He 2024, Chakrabarty et al., 2024, Lu et al., 2025), so more examples are not needed to establish the conclusions of this work.

How "human" text were collected?…How one can guarantee that these texts were not generated?

The Reddit responses are part of the WritingPrompts dataset that was introduced in 2018 by Fan et. al. The models predating 2018 were unable to produce long coherent discourse that constitutes the highly involved stories we collected and studied. The obituaries were from New York Times writers, who we are assuming to be credible human authors.

What parameters (top-p, temperature) were used during LLM-based text generation? It can seriously influence model "creativeness".

We use the default parameters (temperature=1 and top-p=1). We included these details in Appendix B. We experimented with different temperatures in preliminary experiments, but did not observe meaningful differences. This is in line with prior work that found that temperature variation does not reduce the templatic nature of LLM outputs, even at the POS/lexical level (Shaib et al., 2024). In other experiments, we have found that writing quality starts to degrade according to reward model scores when higher temperatures (above 1.3) are used.

Thank you for your detailed feedback and comments! Addressing the concerns raised below:

W1: Why not simply measure tree similarity?

In QUD dependency trees (Wu et al., 2023, Ko et al., 2023), the edge labels are the questions themselves: the edge goes from the answer to the QUD to where the QUD is evoked (or anchored) in prior context. The core idea of QUDsim is to measure cross-document QUD answers, which ultimately comes down to analyzing the QUDs themselves, i.e., the edges of the dependency structures. Only comparing the structures themselves, without the QUDs, will not give us the type of similarity we are looking for.

We do study some notion of global similarity by analyzing ordered QUDs (Section 7). This analysis reveals the existence of templates consisting of QUD sequences. Theoretically, QUD structures can take the form of stacks (Roberts, 2012), trees (De Kuthy et al., 2018), or dependency structures (Wu et al., 2023, Ko et al., 2023). Yet, as a first step towards quantifying similarity with QUDs, we chose a structure-free (i.e., “flat”) option with a level of abstraction that can be operationalized (lines 80-85). The reason for this is to avoid theoretical complexity that can quickly get out of hand: the structures themselves assume some level of entailment relationship between the questions (in the case of stacks and trees), which is beyond the scope of this paper.

W2: Correlation with other metrics?

This work comparatively analyzes QUDsim with three other types of similarity measures which we believe are representative of their kind: (1) N-gram overlap based metrics, from 1-4grams; (2) embedding-based metric, with cosine; (3) LLM-based similarity (see Section 4, “baselines”). ROUGE and METEOR are N-gram based; ROUGE results are worse than 1-4gram baselines included in this work (Rouge-L achieved an F1 score of only 0.026 when computed in the same manner as other values presented in Table 2), thus we did not include it, but we are happy to add to the table. BERTScore is embedding-based, thus will not have divergent results from type (2). (Note that BERTScore itself is highly uninterpretable, with highly clustered numerical values). Our preliminary investigation showed that this is indeed the case, and we can add this to the appendix.

Do you provide an upper bound (u), lower bound (l), or chance estimate (c) for QUDSim? Do these values vary with the length of the discourse?

The upper bound of QUDsim is 1, i.e., when comparing the same document with itself, all answerability scores will be 1. This is shown in the diagonal values of Figure 5. The lower bound of QUDsim is 0, i.e., when no source QUD gets answered in a target document and vice versa. Random pairs of documents in creative writing will also receive values close to 0 since there is little chance that they share the same QUDs; we validated that this is indeed the case with LLM-generated texts randomly sampled from different story prompts. For LLM-generated obituaries of different people, Figure 4 (left) shows the QUDsim values.

Q1: Reverse questions and QUDsim:

This is a very interesting question! Assume that q takes the form “What is the reason for X” and q’ takes the form “What is the result of X”. This means that Q(S,i) and Q(S,j) state the reason for X whereas Q(t,i) and Q(t,j) state the result of X. Since these are different questions, the answerability scores for q in Q(t,i) and Q(t,j) will be 0, and vice versa; thus, there should be no increase in QUDsim. This is a very compelling angle of analysis though, which also speaks to connecting this framework with coherence discourse, and we’d be happy to explore further.

Q2: Do different types of questions (and the underlying discourse dependencies) affect the QUDSim score differently?

We have not conducted an analysis from this angle. Similar to the question above, we also find it interesting, and will be happy to explore further in future work.

Q3: Can you provide a taxonomy of the questions used?

Yes, we would be happy to run the question type classifier from Cao and Wang (2021) and provide the frequency of questions in each type, similar to what was done in Ko et al., 2022.

Suggested citations:

Thanks for the pointers. We will include these in any future version.

Thank you for your feedback and comments! Addressing some of the concerns below:

How do you know where to split the document into segments?

We follow existing text segmentation principles in the LLM-based text segmentation process: “each chunk is about one atomic topic” (Appendix C.2). That said, a first-principles definition for segmentation itself is an open research area (Retkowski and Waibel, 2024 ) In this work specifically, we made the assumption (which was indeed the case from observations) that by using GPT-4o for all queries throughout the pipeline, the LLM makes judgments that are somewhat consistent, so its understanding of granularity may not drastically vary from one document to the next. A full analysis of how granularity impacts similarity is left for future work. Our intuition is that as long as the granularity is fine enough (e.g., paragraph level or a bit below), the relative similarity judgments of pairs of texts here won’t change much.

The granularity of questions in the QUD framework is tricky. [...] how to quantify or set this granularity is undiscussed.

Indeed, QUDs can come at different levels of abstraction. Since our QUDs correspond to the segments (1-2 QUDs per segment, Table 1), which are fine-grained, we avoid having very high-level QUDs. Most QUDs can be answered in just a few sentences, since each segment is at the subparagraph level.

Additionally, our use of entity abstraction helps to decouple fine-grained information from the QUDs used to find similarity. This processing step further reduces the dependence on the precise similarity level. The intrinsic evaluation of this abstraction is in Appendix D.2.

In Figure 2, some annotations are not explained, such as A_Q_k.

Q is the set of all QUDs generated for a given document. Q_k is the k^th QUD in this set. So, A_Q_k is the set of answer sentences for the k^th QUD. We will make this more clear in any future version.

Missing Reference

Thank you! We will include this.