DISRetrieval: Harnessing Discourse Structure for Long Document Retrieval

Dear Reviewer ZveM,

We sincerely thank you for your review and appreciate your recognition that our linguistically motivated tree construction is more intuitive than traditional rigid segmentation approaches. We are happy to address the points raised.

---

Q1: Clarity Issue - How are retrieved results utilized in LLM generation and lack of top-down pipeline overview?

A: We acknowledge that our method description could benefit from clearer exposition. To clarify the complete pipeline:

1. Tree Construction: We parse documents using RST to create discourse-aware hierarchical trees
2. Node Representation: Leaf nodes use embedding models; internal nodes use LLM-generated summaries
3. Retrieval Process: Given a query, we traverse the tree using our structure-aware algorithm (Algorithm 1) to select relevant nodes
4. Generation: Retrieved content is directly concatenated and appended to the query as context for the target LLM

This follows the standard retrieval-augmented generation paradigm used by RAPTOR and other hierarchical methods. We will add a clearer pipeline overview figure in our revision to provide the top-down view that would help readers better understand our approach. And in Appendix D.6, we provide specific prompt structures used for summarization and QA with LLMs. We hope these can help to understand the utilization of retrieval results and LLM generation process.

---

Q2: Missing baselines and analysis - lack of direct document-to-LLM baseline, document length concerns, and computational cost analysis.

A: We appreciate these important concerns and would like to address each point:

Document Length Clarification: There appears to be a misunderstanding about our evaluation setup. The "200-400" refers to retrieved context tokens (following standard evaluation protocols), not document length:

• Retrieved Context: 200-400 tokens (standard evaluation protocol)
• Actual Documents: QASPER papers average 4,170 tokens, QuALITY stories average 5,500 words

Full Document Baseline: We acknowledge your valid point about including full document baselines. To address this concern, we conducted additional experiments comparing different input strategies on QASPER:

These results demonstrate the fundamental principle of retrieval research: targeted retrieval achieves 90% of full document performance while using 10x fewer tokens. The gold evidence results prove that perfect retrieval would actually outperform full document processing, validating the retrieval research paradigm. We will include these comparisons in our revision.

Computational Cost Analysis: Regarding tree construction time:

1. RST parsing uses XLNet (110M parameters) - essentially a single forward pass
2. Tree construction is query-independent and performed once per document
3. In practical applications serving multiple queries, this preprocessing cost is amortized
4. The computational overhead is minimal compared to the 10x efficiency gain in token usage

---

Q3: Transferability concerns - limited transferability and scalability of supervised parsing methods.

A: We understand this concern about domain specificity, but our experimental validation suggests otherwise:

Cross-Domain Validation: We evaluate on two distinct domains - scientific papers (QASPER) and narrative fiction (QuALITY) - demonstrating transferability across vastly different text types with consistent 2-4% improvements across both domains.

Modern RST Robustness: Contemporary RST parsers are trained on diverse corpora and demonstrate robust cross-domain performance. Our consistent results across different text types (formal scientific writing vs. informal narrative fiction) provide empirical evidence against domain-specific limitations.

The systematic improvements across multiple experimental conditions (different models, embeddings, datasets) demonstrate robust algorithmic progress rather than domain-specific optimization.

In Appendix E.1, we provide a systematic analysis of the differences in sentence length and writing style between the two datasets, and demonstrates the adaptive capability of our method in handling different style of documents.

---

Q4: Testing effectiveness on larger scale documents (> 10K tokens)?

A: We assume you mean ">10K tokens" rather than ">10B." This is precisely why we plan to add NarrativeQA evaluation (documents up to 100K tokens) as suggested by other reviewers, demonstrating scalability to extremely long documents. This addition will provide concrete evidence of our method's effectiveness on the larger document scales you're interested in.

---

We believe our discourse-aware approach represents a principled advancement beyond arbitrary chunking methods, and we're happy to discuss further if you have any additional questions.

Best regards,

Authors

Setting: Use 400-length retrieved evidence, wrap corresponding sentences in the full document with [EVIDENCE] [/EVIDENCE] tags (merging adjacent intervals), then use the full document as input to generate results.

Dear Reviewer 5Nq4,

We sincerely thank you for the thoughtful review and for recognizing the strong linguistic grounding of our approach. We are pleased that you appreciate our use of formal discourse theory (RST) and the hierarchical design that moves beyond arbitrary chunking methods.

---

Q1: Clarification on the parameter τ inconsistency in Equation (2) and line 233.

A: We sincerely appreciate you pointing out this critical inconsistency in our notation. This is indeed a significant error that undermines clarity, and we take full responsibility for this confusion.

To clarify the correct interpretation:

• Equation (2): τ represents a threshold parameter for text length comparison, not temperature
• Line 233: The reference to "temperature parameter" is a copy-paste error from our ablation studies on LLM generation parameters

The correct interpretation is: τ is a length threshold where we use direct concatenation when $|t_l|$ + $|t_r|$ < τ, and LLM summarization otherwise.

Our τ=0 for QASPER and τ=100 for QuALITY represent domain-specific optimizations based on document characteristics:

• QASPER (τ=0): Scientific papers benefit from LLM summarization at all levels due to dense technical content
• QuALITY (τ=100): Narrative texts often contain shorter, conversational segments that don't require summarization

We will fix this inconsistency throughout the paper and ensure consistent terminology in our revision.

---

Q2: Concerns about performance degradation on documents that lack clear rhetorical structure or are informal.

A: We appreciate this concern, but our experimental evidence suggests otherwise:

Theoretical Foundation: Rhetorical Structure Theory was developed to analyze all types of coherent discourse, including informal texts, conversations, and narratives. The theory explicitly covers relations like Elaboration, Contrast, and Sequence that appear universally across document types, regardless of formality level.

Empirical Counter-Evidence: Our QuALITY dataset results directly contradict this concern. QuALITY contains diverse narrative styles, including informal dialogue and conversational passages, yet we achieve consistent 3% accuracy improvements across all document types, with no evidence of performance degradation on informal content.

Graceful Degradation: Even if RST parsing encounters difficulties, our hierarchical approach still provides semantic chunking benefits over flat methods, as demonstrated by our consistent improvements across both formal (QASPER) and informal (QuALITY narrative) domains.

Our experimental results suggest that discourse structure benefits extend across document formality levels.

Additionally, Appendix E.1 provides a systematic analysis of the differences in sentence length and writing style between the two datasets, and demonstrates the adaptive capability of our method in handling different style of documents. We hope these results help alleviate your concern.

---

We will also address the minor typos you mentioned (e.g., line 232) through additional proofreading. We believe these revisions will address your primary concerns while maintaining the technical contributions that you acknowledged as good quality and original.

We'd be happy to discuss further if you have any additional questions.

Best regards,

Authors

Setting: Use 400-length retrieved evidence, wrap corresponding sentences in the full document with [EVIDENCE] [/EVIDENCE] tags (merging adjacent intervals), then use the full document as input to generate results.

Dear Reviewer J2Qt,

We sincerely thank you for the comprehensive review and for recognizing our rigorous experimental setup, well-designed ablation studies, and excellent clarity. We are pleased that you acknowledge our method's consistent performance across multiple datasets, models, and embedding approaches.

---

Q1: On the characterization of performance gains as "marginal" - whether the improvements justify the added complexity of discourse parsing.

A: We respectfully challenge the characterization of our improvements as "marginal." The focus on absolute percentage points misses the critical context of retrieval-augmented generation performance:

Retrieval Quality Perspective: Our method achieves 22.63% F1 vs RAPTOR's 20.64% on QASPER retrieval evaluation - a 10% relative improvement in retrieval quality. In information retrieval, such gains are considered substantial.

Generation Upper Bound Analysis: When UnifiedQA receives gold evidence on QASPER, it achieves 51.11 F1-Match. Our method reaches 40.25 F1-Match (78.7% of theoretical maximum), while RAPTOR achieves 38.42 F1-Match (75.2% of maximum). This 3.5 percentage point gap represents meaningful progress toward the theoretical ceiling.

Consistency Across Settings: Unlike methods that show inconsistent gains, our improvements are systematic across all experimental conditions - different models, embeddings, and datasets. This consistency indicates robust algorithmic advancement rather than dataset-specific optimization.

---

Q2: On the limitation to short retrieved context (200-400 words) and utility with larger context windows.

A: This critique fundamentally misunderstands the retrieval paradigm. The goal of retrieval is not to use more context, but to use better context. Our preliminary analysis on QASPER demonstrates this principle:

Gold evidence outperforms full document processing while using 32x fewer tokens. This validates our research direction: targeted retrieval is fundamentally more effective than processing entire documents. Our discourse-aware approach makes meaningful progress toward this gold standard.

Modern LLMs' ability to process thousands of tokens doesn't negate the need for effective retrieval - it makes it more important to identify the most relevant content efficiently.

---

Q3: On RST citations in the introduction and qualitative error analysis of the discourse parser.

A: We appreciate these constructive suggestions:

RST as Exemplar Framework: We want to emphasize that RST serves as one exemplary discourse framework in our work, not the only possible approach. Our contribution lies in demonstrating how any coherent discourse theory can enhance retrieval through linguistic structure. We specifically chose RST due to its formal theoretical foundation and available computational tools.

Qualitative Analysis Already Provided: Regarding qualitative error analysis, Figure 5 provides exactly this type of analysis. It demonstrates how RST parsing quality directly correlates with both retrieval performance and final answer generation. The ablation shows that improved RST model performance leads to better evidence retrieval, which in turn enhances downstream QA performance.

We will expand both theoretical background in the introduction and qualitative analysis, including detailed case studies with visualized structure trees showing how discourse quality directly impacts retrieval results.

---

Q4: On expected performance with less-structured text, such as conversational transcripts.

A: Our QuALITY results provide direct empirical evidence that discourse structure benefits extend to less formal text:

• QuALITY contains diverse narrative styles, including conversational dialogue and informal passages
• We achieve consistent 3% accuracy improvements across all QuALITY document types
• No evidence of performance degradation on informal content

RST was designed to analyze all coherent discourse, not just formal documents. Relations like Elaboration, Contrast, and Sequence appear universally across text types. Even in conversational transcripts, speakers maintain coherent discourse relations when expressing ideas, making our approach applicable.

---

We believe our discourse-aware framework represents a principled advancement in retrieval methodology that extends beyond any single discourse theory, as acknowledged by your recognition of our technical quality, originality, and experimental rigor.

Best regards,

Authors

Setting: Use 400-length retrieved evidence, wrap corresponding sentences in the full document with [EVIDENCE] [/EVIDENCE] tags (merging adjacent intervals), then use the full document as input to generate results.

Dear Reviewer iFfd,

We sincerely thank you for the thoughtful review and constructive feedback. We are glad that you appreciate our linguistically motivated approach and recognize the viability of using discourse structure for tree retrieval methods. We are happy to address the points raised.

---

Q1: Scalability concerns regarding computational overhead and whether modest gains justify the cost.

A: We respectfully disagree with the characterization of "modest gains." To put our improvements in perspective, when we provide UnifiedQA with all gold-standard evidence on QASPER, the upper bound performance is 51.11 F1-Match. Our method achieves 40.25 F1-Match with retrieved context, representing 78.7% of the theoretical maximum - a substantial achievement in retrieval-based QA.

More importantly, while RAPTOR shows only ~1% generation improvement over flat baselines despite its computational overhead from clustering and summarization, our discourse-aware approach delivers consistent 2-4% improvements across multiple settings. From Table 2, our retrieval accuracy improvements are even more pronounced - we achieve 22.63% F1 vs RAPTOR's 20.64% at 400-word context with OpenAI embeddings, representing a 10% relative improvement in retrieval quality.

Regarding computational cost, we emphasize that our discourse parsing overhead is minimal since it relies on XLNet, a relatively small model (110M parameters) where the cost is essentially equivalent to a single XLNet forward pass. More importantly, both discourse parsing and tree construction are query-independent operations performed only once per document. The summarization cost using our adaptive strategy (Equation 2) is also amortized across multiple queries on the same document. In practical applications, this one-time preprocessing cost becomes negligible when serving multiple queries.

---

Q2: Why evaluation is limited to 400 tokens despite larger context windows being available for modern LLMs.

A: We appreciate this important question about context length choices. We acknowledge that our current evaluation setup may not fully demonstrate the potential of our method with longer contexts.

We conducted preliminary experiments comparing different input strategies on QASPER (we focus on QASPER here as QuALITY does not provide gold evidence annotations for this analysis):

These results reveal several crucial insights:

1. Retrieval Potential: Gold evidence consistently outperforms full document input while using 32x fewer tokens (129 vs 4,170). This demonstrates that targeted retrieval is not just more efficient—it's fundamentally more effective than processing entire documents.

2. The Retrieval Challenge: The gap between gold evidence (53.63) and our method (44.52) represents the core challenge in retrieval research—identifying the most relevant content. Our discourse-aware approach makes meaningful progress toward this goal.

3. Practical Implications: In real-world applications, the 32x computational savings of targeted retrieval makes it essential for scalability, even if perfect retrieval remains challenging.

The superior performance of gold evidence over full documents validates our research direction: the key is not processing more content, but processing the right content. Our discourse structure provides a principled way to identify this "right content" more effectively than flat retrieval methods.

---

Q3: Dataset diversity concerns and the suggestion to include NarrativeQA for extremely long documents.

A: We appreciate the suggestion about NarrativeQA. We initially focused on QASPER and QuALITY because they represent two complementary evaluation paradigms: QASPER provides evidence-based evaluation allowing us to assess retrieval quality directly, while QuALITY tests reading comprehension without explicit evidence annotation (similar to NarrativeQA). Since both QuALITY and NarrativeQA lack evidence annotations and cannot be evaluated from a retrieval perspective, we initially chose to include only one representative dataset from this category.

However, you raise an excellent point about document length diversity. While QuALITY documents average 5,500 words, NarrativeQA contains significantly longer documents (some reaching 100k+ tokens), making it valuable for testing scalability. We will add NarrativeQA experiments in our revision to demonstrate how our discourse-aware approach performs on extremely long documents, complementing our current evaluation scope.

---

Q4: Technical question about τ=0 for QASPER - why LLM is always used to summarize.

A: This is an excellent technical question. We set τ=0 for QASPER because scientific papers benefit from LLM-enhanced summarization at all levels due to their dense technical content. For QuALITY (τ=100), narrative texts often contain shorter, more conversational segments that don't require summarization.

This adaptive strategy is validated by our ablation studies showing optimal performance with these settings. The domain-specific optimization reflects the different linguistic characteristics between scientific and narrative texts, where scientific content consistently benefits from abstraction while narrative content can often be used directly when segments are sufficiently short.

---

We believe these additions will address your concerns while maintaining the core contributions that you acknowledged as technically sound and original. We'd be happy to discuss further if you have any additional questions.

Best regards,

Authors

Setting: Use 400-length retrieved evidence, wrap corresponding sentences in the full document with [EVIDENCE] [/EVIDENCE] tags (merging adjacent intervals), then use the full document as input to generate results.