DocKS-RAG: Optimizing Document-Level Relation Extraction through LLM-Enhanced Hybrid Prompt Tuning

We sincerely thank the reviewer for recognizing and affirming our work. Regarding the questions raised during the review process, we have carefully considered them and provided detailed responses as follows:

Q1: In Introduction, the authors state that “graph-based methods … lack of sufficient contextual information”. The authors need to explain why they believe graph-based methods lack contextual information and what specific types of context are missing.

Response to Q1: Thank you for your valuable question. Graph-based methods primarily focus on the structural relationships between entities, which can obscure essential semantic nuances due to the static nature of graph representations. In complex document-level relation extraction tasks, entities may relate across texts, paragraphs, or even chapters within a document. This limitation in the graph-based approach restricts our understanding of how entities interact in different contexts, ultimately hindering the effectiveness of extraction.

Q2: Also in Introduction, the authors state that “PLMs are … the semantic misalignment between the texts and graphs”. I don’t understand how they address this issue in this paper.

Response to Q2: As shown in Section 4.4, we integrate informative structural knowledge from DocKG with the relevant sentences retrieved by SetRAG, and further generate the hybrid prompts to enhance the interaction between the textual data and the graph-based representations, thereby improving relation extraction performance.

Q3: Where do the training samples come from? Besides, will this process limit the applicability of the method to other domains?

Response to Q3: Thank you for your insightful questions. The training samples for constructing DocKG come from observable documentation data, which is utilized to obtain representations of entities and relations. In our proposed DocKS-RAG framework, the training process entails the construction of DocKG and SetKB. Relevant subgraphs and sentences are retrieved to create hybrid prompts, which are then fine-tuned using PEFT to improve the performance of LLMs in document-level relation extraction tasks. Although DocKS-RAG relies on this training process, it is adaptable and can be applied to other domains by fine-tuning the model with domain-specific data.

Q4: In Eq.(10), the authors introduce a generation function $g_{DocKG}$. How do they implement this function?

Response to Q4: Thank you for your thoughtful insights. $g_{DocKG}$ is implemented as a function for graph knowledge transformation. We pre-defined various entity-relation mapping types. For example, if a subgraph has "Wisconsin" and "U.S." connected by the edge "state", it would be transformed into the statement "Wisconsin is a state of U.S." based on our mappings. Ultimately, we concatenate such statements with the relevant semantic information retrieved from SetRAG, and obtain the hybrid prompts.

C1: When explaining the extraction of knowledge graphs from the knowledge base, the authors mention the user query q. However, in Section 3 (Problem Definition), there is no indication that a user query is provided, and in Figure 1, the query is also not visible. The authors need to clarify this inconsistency.

Response to C1: As mentioned in Section 4, q is denoted as the input user query for further extraction. We appreciate your insight and will consider including the explanations about q in the Problem Definition section and Figure 1 in the revised manuscript to enhance the readability.

C2: In Section 5, the authors provide the hyper-parameter experiments before the ablation study, which seems to be weird for me. I suggest rearranging the order to improve clarity and coherence.

Response to C2: Thank you for your valuable suggestion. We will consider making this change to the revised version.

Thank you for your valuable feedback.

C1: Although the paper provides a detailed methodology, some sections could benefit from greater clarity. For instance, the hybrid prompt generation process may require more concrete examples to fully illustrate the differences and advantages over conventional methods.

Response to C1: We appreciate your feedback, particularly regarding the clarity of the hybrid prompt generation process. We aim to enrich this section by including specific examples that illustrate how the generated prompts differ from those produced by conventional methods. By showcasing practical scenarios, we intend to clarify the advantages of our hybrid approach and its effectiveness in providing richer context for relation extraction tasks.

Q1: Could you provide a detailed explanation of the operations of Formula 10 and Formula 15?

Response to Q1: Thank you for your question. As illustrated in Section 4, the purpose of Formula 10 is to generate informative prompts based on the retrieved entities and relations from DocKG. It defines the prompt generation function $g_{DocKG}$, which takes a set of relevant entities and relations as input and outputs structured prompts $P_{DocKG}$​. Formula 15 constructs hybrid prompts using the retrieved sentences from SetKB. It employs the generation function $g_{SetRAG}$​ to create prompts from the relevant triplets $T_k$​ while appending the retrieved sentences $S_k$​. This integration enriches the prompts by combining contextual insights with structured knowledge, enhancing the model's ability to perform document-level relation extraction accurately.

Q2: In your results, you mention challenges related to semantic misalignment between PLMs and knowledge graphs. Can you elaborate on how your framework specifically addresses this issue?

Response to Q2: Thank you for your thoughtful insights. As mentioned in Section 4.2, the embedding spaces of PLM and DocKG differ significantly. To address this issue, we integrate PLM embeddings as the initial inputs for entities and relations, which allows DocKG to learn from the PLM during the updating process and improves the reliability of similarity calculations.

Q3: Could you clarify the process for generating these hybrid prompts, and perhaps provide examples of how they differ from standard prompts?

Response to Q3: Thank you for your insightful feedback. As shown in Section 4.4, we incorporate both relevant structural and semantic information from DocKG and SetRAG. Specifically, we pre-defined various entity-relation mapping types. For example, if a subgraph has "Wisconsin" and "U.S." connected by the edge "state", it would be transformed into the statement "Wisconsin is a state of U.S." based on our mappings. Ultimately, we concatenate such statements with the relevant semantic information retrieved from SetRAG, and obtain the hybrid prompts.

Thank you for your positive comments and insightful questions.

Q1: While DocKS-RAG achieves high performance, the ablation studies suggest that even without blending structural and semantic components, competitive results can be obtained. For example, simpler setups without DocKG or SetRAG achieve reasonably good scores. Could the authors elaborate on the trade-offs between complexity and efficiency in practical deployments?

Response to Q1: We appreciate your insightful question. In practical applications, we take full advantage of the parallel processing capabilities provided by both DocKG and SetRAG, enabling us to handle tasks efficiently while maintaining high performance of relational extraction. Additionally, by applying threshold retrieval strategies, we could further simplify the complexity of our operations.

Q2: In your discussion, you mentioned that existing graph-based methods can introduce noise and irrelevant connections, affecting performance. Can you elaborate on how DocKS-RAG specifically mitigates this noise during knowledge graph construction and retrieval processes?

Response to Q2: Thank you for your valuable feedback. As presented in Section 4, firstly, DocKS-RAG applies a threshold parameter $\tau_{er}$ during graph retrieval to filter out extraneous entities. Secondly, another threshold parameter $\tau_{eq}$ is used to retrieve the relevant contextual sentences. Finally, DocKS-RAG generates hybrid prompts that combine the information from both the DocKG and SetRAG, which integrates relevant contextual information to enhance semantic alignment, improving the overall extraction performance.

Q3: Regarding future work, how do you envision scaling DocKS-RAG to accommodate different types of documents or domains with vastly different structure or language use (e.g., legal texts, scientific literature)? What modifications might be necessary to adapt the framework effectively?

Response to Q3: To scale DocKS-RAG for diverse document types or domains, modifications could include tailoring the DocKG construction to account for domain-specific terminology and relationships, alongside enhancing the SetRAG mechanism to accommodate varied syntactical structures and language use. Additionally, training additional domain-specific models or fine-tuning existing ones with targeted datasets would improve adaptability and semantic accuracy for specific contexts.

Thank you for your positive feedback on our paper.

C1: When mentioning the sentence-level Semantic Retrieval-Augmented Generation, it might be useful to briefly explain how it differs from typical retrieval mechanisms.

Response to C1: Thank you for your valuable comment. Our proposed SetRAG module constructs a knowledge base from segmented sentences and retrieves contextually relevant information based on embeddings, allowing for improved understanding of the document's semantics. Such semantic information will be integrated with structural knowledge from DocKG, which further enhances the performance of LLMs in relation extraction tasks.

Q1: Although DocKS-RAG demonstrates high performance, the ablation studies indicate that competitive results can still be achieved without integrating structural and semantic components. For instance, simpler configurations that do not utilize DocKG or SetRAG yield reasonably good scores. Could the authors elaborate on the trade-offs between complexity and efficiency in practical deployments?

Response to Q1: Thank you for your thoughtful question. As shown in Table 4, the experimental results indicate that the integration of the DocKG and the SetRAG mechanisms significantly enhances model performance, demonstrating the effectiveness of our proposed DocKS-RAG module. In practical deployment scenarios, we capitalize on the parallelization capabilities offered by both DocKG and SetRAG, which allows for efficient processing without compromising the quality of relational extraction. Furthermore, by implementing strategies such as threshold retrieval, we manage to streamline the complexity of the operations involved. Integrating both modules enhances the model's adaptability to various document types and ensures robust performance in real-world applications.