Faithful Rule Extraction for Differentiable Rule Learning Models

We thank the reviewer for the valuable comments.

It is not clear to me why faithfulness is an important metric for rule learning. What kind of benefits can we obtain for the rule learning problem if the rules can respect faithfulness, e.g., say better generalizability? The authors may need more effort to justify this. One motivating example might be helpful for readers to understand this. ...One suggestion would be using some running examples to intuitively explain the meaning of the notations and equations rather than just leaving them in the paper.

As requested, we have incorporated an example in the introduction, highlighting the benefits of ensuring faithfulness. We now also refer to this running example in other parts of the paper. Given the tight space restrictions and the need to rigorously present our technical results, this description is necessarily concise. Nevertheless, we anticipate that it will enhance the overall readability of the paper.

I also feel that the proposed solution is too specific. It is only applicable to the DRUM model rather than the general rule learning model, which may limit the applicability of the proposed method in general settings.

While it is necessary to independently analyse the faithfulness of various rule-learning models, we expect that the strategies employed in our approach will contribute to a broader understanding of the faithfulness of rule-learning models. Indeed, we anticipate that these strategies can serve as a foundation for the future analysis of other models.

The authors only focus on evaluating the performance of the proposed methods. However, no appropriate evaluations (except some simple case studies) are conducted for the faithfulness part, e.g., how many rules from the vanilla DRUM model violate the faithfulness criterion and what is the bad effect of including those rules?

Please note that in Section 6 we perform an extensive analysis of the faithfulness of the rules extracted from the original DRUM model (results are summarised in Table 2). The soundness of the extracted rules is guaranteed by Theorem 1. However, Table 2 shows that the rules are not complete, and in fact they only produce a very small fraction (less than 7%) of the facts predicted by the DRUM model. Therefore, none of the rule sets extracted from the original DRUM model is faithful.

We thank the reviewer for the comments and suggestions. We have uploaded a new version of the paper including the references provided by the reviewer.

We thank the reviewer for the valuable comments. We answer the questions raised by the reviewer as follows.

As best as I can tell, the techniques in the paper do not deal with classification / question answering / recommendations.

The techniques presented in the paper address the general problem of realising transformations between KGs. Tasks such as classification, question answering, and recommendation on KGs can be seen as concrete applications of our approach. Question answering over KGs can be viewed as the problem of learning a function that maps the input knowledge graph to a set of facts representing all answers to a given set of questions. Recommendation can be viewed as learning a function that maps a KG describing user-item interactions to a set of facts where each fact represents a recommendation. Finally, node classification can be seen as the problem of learning a function from a KG to a set of facts stating membership of nodes to given classes.

It is not obvious (either from the description of the technique or the experimental results) that they provide assurances around the "precise relationship between the model and the extracted rules" (while there might be soundness, completeness does not seem to result).

Please note that Theorem 3 in Section 4 ensures the faithfulness (i.e., both soundness and completeness) of Algorithm 1 for multipath rule extraction from a DRUM model. Furthermore, Theorem 4 in Section 5.1 proves the faithfulness of Algorithm 2 for multipath rule extraction over a fixed dataset. The corresponding proofs are given in the Appendix.

We thank the reviewer for the comments and for pointing out typos and other minor infelicities in the references, which we have fixed in the updated version of the paper.

This paper is not written for an audience outside this immediate community, and the authors also do not make any attempt to make their work more accessible. If you do not know any of the prior works, in particular DRUM, you are lost. I could, e.g., easily follow the terse formal description of Datalog, because I am familiar with that. With the similarly formal presentation of DRUM I was completely lost. I now know what parts this system has, but not what they are used for. I know that path-counting is part of the method, but I don't know what paths are counted, and why this has any significance.

To make the paper more accessible, we have uploaded a new version with running examples that illustrate the key ideas and models discussed in the paper. The new examples are introduced in Section 1 and Section 3 (after Lemma 1), respectively, and we expand upon them in the following sections. Given the tight space restrictions and the need to rigorously present our technical results, the discussion of the examples is necessarily concise. Nevertheless, we anticipate that they will enhance the overall readability of the paper.

I'm not really familiar with this task, but this seems to be quite a heavy machinery for extracting family relations. In Inductive Logic Programming these are only toy examples. Are there no better tasks for showcasing what you can do?

We used the Family dataset in Section 6 to illustrate some simple rules extracted by our model which are intuitive and easy to read. We have now included in the Appendix examples of rules extracted for FB15k-237, WN18RR and NELL-995, which are real-world datasets commonly used as KG completion benchmarks.