Bio-RFX: Refining Biomedical Extraction via Advanced Relation Classification and Structural Constraints

We appreciate your constructive feedback, which has been instrumental in enhancing our paper. We have addressed all the points raised and have detailed our responses below.

Weaknesses

The approach could be generally applied to other domains, but the approach is presented as a model for the biomedical domain, and the scope is limited.

This approach is designed to tackle specific issues in biomedical texts. The relation-first approach is based on the high prevalence of ambiguous terms in biomedical literature and serves as a hint for the entity type. The implementation of structural constraints is informed by domain-specific knowledge. For instance, the relation activator can only occur between a chemical and a gene. On the other hand, the relation located_in can occur between sports_team and city, company and country, city and country, among others, thus offering only a very weak constraint. We will explore the potential of applying our approach to other domains in our future research.

The comparison with existing state-of-the-art entity relation models is limited. The authors presented several approaches like OneRel and SPN in the related work section, and there are several SOTA models for entity relation tasks as follows, but the comparison is not performed.

Inspired by your constructive comments, we have updated the related work section in the revised version of the paper. We will further conduct comprehensive research into these SOTA models, extending them into biomedical fields and performing in-depth comparisons and discussions.

Questions

Please see the weaknesses above.

Kindly refer to our reply to Weakness 1 and Weakness 2.

How is the model specific to the biomedical domain?

Please review our response to Weakness 1.

Appendix A shows several comparisons of prompting using domain resources, but the prompts used in practice are not clear. For the activator relation type, the authors use the prompt including activate (not activator), so is question generation done manually? They also say, "Note that it is a relatively simple approach" in explaining questions, but do they use any other complicated approach in practice? Or is this simple approach the best?

In practice, we exclusively use the questions and refrain from using any prompts derived from domain resources.

All the questions are generated based on templates except for entity detection (Section 3.2.1) in relation extraction task. The primary reason is that the relation types in different datasets are morphologically and semantically diverse, and manual generation is more accurate in terms of syntax and semantics. In our future work, we plan to investigate techniques for automatically generating questions using the appropriate toolkit. While for the other components in our model, the question is generated based on templates. For example, in number prediction, the template is How many $\tau_{e1}, \tau_{e2}, ..., \tau_{eN}$ are there in the sentence with relation $\tau_r$?, where $\tau_{e1}, \tau_{e2}, ..., \tau_{eN}$ are all the entity types that satisfy the structural constraint of relation $\tau_r$.

In Appendix A we also mentioned two prompting techniques: term definitions and UMLS markers. Both of them have a negative influence on the model's performance.

Term Definition. We enrich the question with definitions from the Free Medical Dictionary (https://medical-dictionary.thefreedictionary.com/), i.e. Bio-RFX (+definition). In NER, the question is followed by the definitions of all the entity types that appear in the question. Similarly, for RE, definitions of the present relation type. The experimental results are displayed below. The rigid definitions introduce noise into the data distribution, thereby increasing the complexity of modeling sentence representations.

UMLS Marker. As suggested in Appendix A, incorporating UMLS markers leads to a performance drop, due to the discrepancies between entity types of UMLS Metamap and datasets, the low accuracy of Metamap matching, and the overlooking of relation types in the sentence. In the following example, the term of is erroneously identified as a gene OF (TAF1 wt Allele) due to its ambiguous nature, which subsequently hampers the overall performance. Moreover, accessing Metamap via web API is extremely time-consuming, posing a challenge to processing speed.

... <CHEMICAL> isoprenaline </CHEMICAL> - induced maximal relaxation ( E ( max ) ) <GENE> of </GENE> <CHEMICAL> methacholine </CHEMICAL> - contracted preparations in a concentration dependent fashion ...

We appreciate your insightful suggestions and apologize for any lack of clarity in our paper. We have addressed all the issues, and the detailed responses are as follows.

Weaknesses

The choice of baselines seems arbitrary; I suggest linking Section 2.1 and Section 4.2.1 to make the reason for "why cannot use relation-first baseline" more explicit.

In Section 2.1, we introduce two relation-first methods: PRGC and RERE, both of which are not suitable baselines for our task.

PRGC only performs the relation classification task, which means that it uses ground truth entities as the input and predicts the relations between entity pairs. On the contrary, our method is able to extract both entities and relation triplets.

RERE extracts subjects and objects from the text, ignoring entities that have not been found in any relation triplets, while our approach can recognize all the entities in the text. Besides, RERE does not differentiate between multiple mentions of the same entity within the sentence. However, biomedical literature is seethed with complicated clauses and ambiguous terms, which necessitates the precise identification of each unique mention. Thus, the benchmark is designed to distinguish each mention during the process of metric calculation, and RERE has to be excluded from baselines.

It isn't easy to gain insights into the main strengths of the proposed method. See question A.

Please refer to our response to question A.

Questions

It is unclear what the ablated variant "- Structure" is. Suggest linking Section 4.5 and the four components in Section 3. For example, the "- Number" variant contains only component 1, 2, 4? and, it would be nice to see a variant containing only the first two components.

Referring to Section 3, our framework contains four key components:

1. Relation Classifier
2. Entity Span Detector
3. Entity Number Predictor
4. Pruning Algorithm

Apart from the four key components mentioned above, we take advantage of the structural constraints brought by domain knowledge to obtain more accurate relation triplets.

In addition to the brief explanation on page 9 of our revised paper, we present a more detailed discussion here. In order to measure the significance of the structural constraints, we performed the ablated variant "- Structure" by removing the structural constraints informed by domain-specific knowledge from the model. Without taking advantage of the structural constraints, the model will end up obtaining less accurate relation triplets. For example, the relation activator can only occur between a chemical and a gene. The ablated variant without the structural constraints may extract relation triplets containing relation activator and two gene's.

In order to test the validity of the entity number predictor, we performed the ablated variant "- Number" containing only components 1, 2, and 4, for which we use the average number of entities in a sentence as the threshold for the pruning algorithm during inference.

We have performed an ablated variant containing only the first two components, i.e. the relation classifier and the entity span detector. While the micro F1 score for NER on the DrugProt dataset increased by 1.35%, the micro F1 score for RE on the same dataset decreased dramatically by 4.58%. This demonstrates that although the ablated variant may recall slightly more entities, it will extract a lot more false relation triplets due to the existence of perplexing entities, thus harming the precision.

Thank you for your prompt feedback. As outlined in our previous response under Weakness 1, a fair comparison between Bio-RFX and relation-first baselines is not feasible. We appreciate your understanding and look forward to further discussions on this matter.

Thank you for the constructive comments. They are very helpful for improving our paper. We understand your concerns and would like to address them as follows.

Weaknesses

I was unable to find information about statistical analysis (e.g., statistical significance tests or confidence intervals).

Thank you for your constructive suggestions. We are conducting more experiments regarding statistical tests according to your new comment and will update the results before November 22.

Automatic entity and relation extraction is a trending topic in research on natural language processing, knowledge graphs, and machine/deep learning. Hence, a more convincing case for novel contributions made in this paper could be made. For example, I could not find a single paper on contrastive representation learning included in the paper, although, e.g., triplet loss and contrastive representation learning are very closely related (see, e.g., Le-Khac, P. H., Healy, G., & Smeaton, A. F. (2020). Contrastive representation learning: A framework and review. IEEE Access, 8, 193907-193934).

Thank you for your comments, which have broadened our perspective. We will conduct in-depth research and discuss it in detail in our subsequent work.

Reference list of the paper could be perfected and math should be punctuated.

Thank you for your detailed and helpful suggestions. We have carefully polished the paper and fixed the writing issues in the revised manuscript.

Questions

How were the performance gained evaluated as significant? Were they both statistically and practically significant?

Please see our response to Weakness 1.

What made the contributions made new/novel compared to prior work?

Compared with other methods, our approach introduces novelty in the following aspects. Firstly, we take advantage of the structural constraints brought by domain knowledge to obtain more accurate relation triplets. This approach mitigates the issue of extracting false positive triplets from the texts when enumerating all the entity pairs. Secondly, in order to address domain-specific issues, such as nested or overlapping biomedical terms, we implemented the text-NMS algorithm to improve the specificity of extraction. Thirdly, we generate a question query with respect to the relation type and targeted entity type, providing an intuitive way of jointly modeling the connection between entity and relation.

It would have been helpful to communicate the results from those additional statistical tests in the rebuttal already. Because these results - or even methodological detail towards conducting the tests - were not included, I am not in a position of changing my review for the better. Based on the other reviewers' concerns, I have now revised the review to go from marginally above to marginally below the acceptation threshold.

Thank you for your prompt response. We have designed a statistical analysis and performed experiments on the DrugVar dataset according to your constructive suggestions, which we address as follows.

1. We choose 5 seeds randomly.

2. We train Bio-RFX and all the baseline models with each seed and record the corresponding performances.

3. We perform one-tailed paired t-tests between Bio-RFX and each baseline model with significance level $\alpha = 0.05$ on the results. For each baseline model:

   1. We compute the difference in performance between Bio-RFX and the baseline model so that we obtain 5 difference measures $d_i~(i = 1, 2, \dots, 5)$.

   2. We compute the $t$ statistic under the null hypothesis that Bio-RFX and the compared baseline have equal performance:

      $$t = \frac{\bar{d} - 0}{s / \sqrt{5}} = \frac{\sqrt{5}\bar{d}}{\sqrt{\frac14\sum_{i=1}^5(d_i - \bar{d})^2}},$$

      ​ where $\bar{d}$ and $s$ are the sample mean and standard deviation of the difference measures, respectively.

   3. We compute the p-value and compare it to the significance level $\alpha = 0.05$. If the p-value is smaller than $0.05$ or the $t$ statistic is bigger than $2.132$, we reject the null hypothesis.

The $t$ statistics and p-values between Bio-RFX and the baseline models are shown in the following table. We can observe that all the p-values are below $\alpha = 0.05$ (and all the $t$ statistics are above $2.132$), rejecting the null hypothesis and demonstrating that Bio-RFX significantly outperforms all the compared baselines. We hope these results may alleviate your concerns. We have updated the experimental results, please refer to our latest response.

We have further conducted significance tests using the Bacteria Biotope dataset, and we present here our experimental results on the Bacteria Biotope dataset. It shows that Bio-RFX significantly outperforms the baselines on this dataset too. Due to our limited computing resources and time constraints, we may not be able to conduct the significance tests for the DrugProt dataset before Nov. 22, as it is a much bigger dataset, instead, we will see if we can perform the experiments for the DrugProt dataset in the low resource setting (i.e., with much smaller training data) by the due date. In future versions, we will carry out more comprehensive significance tests and analyses. We appreciate your understanding and patience.

We have conducted significance analysis for the DrugProt dataset in the low resource setting (i.e., with a much smaller training set of size 200). The results are shown below. We would like to highlight that even when training resources are limited, our method is still able to demonstrate significant performance improvements over the most of the baselines. We appreciate your time and consideration, and we look forward to any further comments or suggestions you may have to enhance our work.

Thank you for the valuable and detailed comments on the experimental results, which are very helpful for improving our paper.

Questions

Looking at the results in Table 1: What is the hypothesis for KECI performing better on RE task in DrugProt?

In addition to the brief explanation on the last line of page 7, we present a more detailed discussion here. The result in Table 1 can be attributed to two key factors: the integration of a background Knowledge Graph (KG) within KECI, and the large number of training samples in the DrugProt dataset.

KECI first constructs an initial span-graph from the text and then uses an entity linker to form a knowledge graph containing relevant background biomedical knowledge for the entity mentions in the text. To make the final predictions, KECI fuses the initial span graph and the knowledge graph into a more refined graph using an attention mechanism. The intricate network of interconnections among entities within the background KG equips KECI with a deeper understanding of sentence structures, thereby facilitating the formation of complex triplets. On the other hand, in order to lessen the demand for computational resources, we match all the possible subjects and objects to generate triplets, as stated in Section 3.2.3.

Another contributing factor is the larger size of the DrugProt training set compared to other datasets, as detailed in Table 5 in Appendix C. This allows KECI to effectively strike a balance between utilizing training data and incorporating prior knowledge. However, when training data is limited, it will lead to overfitting to the background KG. For additional supporting experimental results and discussions, please refer to Section 4.4.