MMD-NSL: Mixed Multinomial Distribution-based Neuro-Symbolic Learning

Thank you for your helpful comments and questions.

Response to Weakness 1:

We have added and discussed references [1]-[6] to the related works section, in section of related work in red.

Response to Comment 1:
We mean Horn clauses form, and we have revised it in Section 3.1, first paragraph, with a red mark.

Response to Comment 2:
We have clarified it in Section 3.1, first paragraph, with a red mark.

Response to Comment 3:

We have revised it in Section 3.1, first paragraph, with a red mark.

Response to Comment 4:
We have revised Definition 1 for clarity in revision version. All embeddings, including {eNERhead , eNERtail , epath}, are explicitly defined and used within these functions. Their roles and how they are obtained are explained in detail as part of the definitions of the respective functions.

Response to Comment 5:
We have corrected this typo in Remark 3, marked in red.

Response to Comment 6:

The categorization may vary depending on different interpretations. For instance, in the highly cited survey From Statistical Relational to Neuro-Symbolic Artificial Intelligence, LTN is categorized as extending MLNs, alongside Neural Markov Logic Networks (NMLN) and Relational Neural Machines (RNM). Specifically, the survey states:

"Another group of approaches, including Logic Tensor Networks (LTN) [Donadello et al., 2017], Neural Markov Logic Networks (NMLN) [Marra and Kuželka, 2019], and Relational Neural Machines (RNM) [Marra et al., 2020], extend MLNs, allowing either predicates (LTN) or factors (NMLN and RNM) to be implemented as neural architectures."

In contrast, PSL [7], which you mentioned, is categorized in the survey under a parallel class of Markov Logic Networks (MLNs), as the survey states: "The second category includes Markov Logic Networks (MLNs) [Richardson and Domingos, 2006] and Probabilistic Soft Logic (PSL) [Bach et al., 2017]. They essentially specify a set of weighted constraints, clauses or formulae."

Response to Question 1:

We have added a figure (Figure 1) in the main paper, highlighted in red, to illustrate an overview of the proposed framework. This figure explains the handling of long dependency chains and complex categorization through a detailed example, as described in the red-marked text within the introduction section.

Response to Question 2:
The formulas have been updated with changes marked in red.

Response to Question 3: The known facts refer to rule samples (⟨C(h), r_head, C(t)⟩ → r1 ∧ . . . ∧ rL) in the KG training dataset. Whether a rule sample holds true indicates the ground truth label for the relation r_head under the context C(h)-C(t), specifying whether it is true (positive) or false (negative) based on the given rule body. All relations in the rule are quantified by relation embeddings, while NER types are quantified by NER embeddings. These rule samples are utilized in Eq.4, with further explanation text provided below Eq.4.

Response to Question 4:

Equation 1 can unify traditional MLNs with their extended variants. Specifically:

1. Pure MLNs: $f_j$ corresponds to $n_j$, the number of rule groundings, softened by a weight $w_j$.
2. Fuzzy-Based MLN Extensions: $f_j$ represents fuzzy operators, softened by a fuzzy score weight $w_j$.

The inputs to $f_j$, which can include rule chains or logic rule formulas, are collectively denoted by $z_j$.

Response to Question 5:

Context Logic Semantic Variable in Eq.2 is inspired by the Dot-Product Attention mechanism. Consequently, in Theorem 2, the attention weights approximate the Context Logic Semantic Distribution composed of the Context Logic Semantic Variable.

Fuzzy Logic Semantic Variable is inspired by fuzzy conjunction and fuzzy disjunction operations. These variables serve as a key link for establishing a bijective mapping between the MLN-based logic semantic distribution and the multinomial distribution-based logic semantic distribution in Theorem 1.

Therefore, these two types of variables serve our paper’s aim of addressing context categorization and long dependency modeling. They form a necessary foundation for constructing a logic continuous distribution based on a mixed multinomial distribution, as discussed in Theorem 3.

Our ultimate goal is to construct a unified logic continuous distribution based on a mixed multinomial distribution. This unified framework is achieved through the combination of the two types of Logic Semantic Variables.

Other forms of Logic Semantic Variables could also be considered, provided they can be proven to form a mixed multinomial logic distribution.

We respectfully request reconsideration of our score.

Thank you for your helpful comments and questions.

Weakness 1: The presentation overall lacks clarity and needs further refinement. This includes many grammar errors and issues such as exact repetitions within the same page (e.g., "establishing a bijective mapping between MLNs and continuous multinomial distributions," both on lines 66 and 76).

Response to Weakness 1:

In revision version, we have revised significant portions of the introduction to eliminate redundant sentences. Additionally, we have added a new figure (Figure 1) to visually explain the overall framework, improving clarity and addressing the need for further refinement. These changes are highlighted in red within the revised introduction section.

Weakness 2: The Figures do not support the significance of the contribution well. Figure 1 takes up half a page to demonstrate that a simple synthetic target could be learned. Figure 2 takes up an entire page with a questionable amount of information for the reader. Also, in Figure 2, labels are badly formatted (overlapping numbers) with misleading and inhomogeneous color mapping (e.g., a difference of 0.01 changing the hue completely). Again, some phrases are nearly identical, e.g., between Figure 1 caption and lines 442-446.

Response to Weakness 2:

In revision version, we have removed the Synthetic Experiment section to allocate more space for detailed explanations of real-data experiments. Regarding the visualization in Figure 2, we have re-drawn the figure to address the issues you highlighted, including overlapping labels, misleading color mapping, and inhomogeneous formatting. Additionally, we have completely revised the section on rule probabilistic visualization to provide a more thorough explanation of the learning rules and their significance, ensuring that the updated figure effectively supports our contributions.

Weakness 3:
Given the nature of this contribution, namely a novel NeSy learning architecture, it would be highly desirable to compare performance across multiple datasets instead of a single one.

Response to Weakness 3:
We would like to emphasize two key points:

1. Our contribution is not solely focused on performance comparison with a single dataset. Instead, we evaluate using 23 NER-pair-aware datasets. Those fine gained dataset provides a rigorous evaluation approach.
2. Our research aims to better model complex NER-pair categorization rather than traditional long dependency considering baselines. Most used datasets do not offer sufficient meta-information to refine the data for rule-based NER-pair research, which is crucial for modeling complex NER-pair categorization effectively.

Weakness 4:
The mathematical presentation is, at times, lacking. For example, in the introduction and overview sections, logical rules are presented as a head implying the body (r_head -> r_1, r_2, ..., r_n). Traditionally, the opposite is the case, with r_head <- r_1, r_2, ..., r_n being the way Horn Clauses are written in implication form.

Response to Weakness 4:

It's a typo of arraw symbol, and we have revised it in revision version, : 'NSL tasks typically consider long dependencies expressed in the form $r_{\text{head}} \leftarrow r_1^j \wedge \ldots \wedge r_L^j$ , where $j$ represents the $j$-th unique rule body for the current $r_{\text{head}}$, and $L$ is the length of the relation along a rule body path.' in Section 3.1, first paragraph, with a red mark.

We respectfully request reconsideration of our score.

Thank you for your helpful comments and questions.

Weakness 1: Ontology information is not fully utilized in the framework; it should also be applied to the tail part of the rule. This enhancement would make the paper more comprehensive and increase its contribution.

Response to Weakness 1: Our framework inherently addresses this concern through Theorem 2, which demonstrates that attention mechanisms can simultaneously encode ontology information for both the rule head and the rule body using a transformer architecture. Specifically:

• Self-attention encodes ontology information related to the rule head.
• Encoder-decoder attention further integrates the ontology information of the rule head while also encoding the rule body.

This dual capability eliminates the need for explicit extensions to the rule tail, as the framework already captures and integrates ontology-related information effectively. By leveraging these mature and well-established mechanisms, our approach ensures comprehensive utilization of ontology information without additional modifications to the framework.

Weakness 2: The experimental design lacks a key analysis: how the two components—the ontology information and the MLN-based probabilistic framework—individually contribute to the observed improvements.

Response to Weakness 2: In Table 1, we have implemented and compared two baseline methods: one based purely on multinomial distributions without incorporating ontology information, and another using a purely MLN-based approach. These comparisons clearly demonstrate the individual contributions of each component, highlighting the observable improvements achieved by integrating ontology information and the MLN-based probabilistic framework within our method.

Weakness 3: Certain aspects of the algorithm are unclear. For instance: 1) How are the rules generated? 2) How is n_j calculated for each newly generated rule? The paper addresses some important points in this field but does not yet meet the standards for the conference.

Response to Weakness 3:

In revision version, we have added a description of rule generation and the calculation of $n_j$ in Section 3.4 (Algorithm), highlighted in red.

We respectfully request reconsideration of our score.

Thank you for your helpful comments and questions.

Weakness 1: Lack of Comparisons with More Diverse Baselines: The paper does not benchmark against a sufficiently broad spectrum of established baselines, which is essential for assessing relative performance comprehensively.

Response to Weakness 1:

The core principle of neuro-symbolic learning lies in mapping symbolic reasoning to various continuous spaces through differentiable methodologies tailored to specific settings and tasks. Our work more focuses on unifying and establishing bijections across different continuous mapping methods, providing a robust theoretical framework supported by strong mathematical underpinnings.

As mentioned in Section 5.2, we carefully selected three representative approaches that are closely tied to our Theorems and Remarks. For instance, Theorem 1 states: "There exists a fuzzy logic semantic function $F_{\text{rule}}(z_j)$, in the form given in Eq.3, that establishes a bijective mapping from the MLN-based logic semantic distribution in Eq.1 to a multinomial distribution-based logic semantic distribution."

These three categories of approaches include:

1. LTN [1], which represents fuzzy logic-based continuous mapping and is used to evaluate Theorem 1.
2. NMLN [2], a neural network-based MLN continuous mapping method that can evaluate Remark 2.
3. RNNLogic [3] and LogiRE [4], which utilize multinomial distribution-based logic mapping and can evaluate Theorem 3.

We believe that conducting theorem-driven ablation experiments is a more meaningful and rigorous strategy than simply stacking numerous empirical evaluation baselines, as it directly validates the foundational contributions of our framework.

[1] Ivan Donadello, Luciano Serafini, and Artur S. d’Avila Garcez. Logic tensor networks for semantic image interpretation. In Carles Sierra, editor, IJCAI, 2017. [2] Giuseppe Marra and Ondrej Kuželka. Neural markov logic networks. CoRR, abs/1905.13462, 2019. [3] Qu M, Chen J, Xhonneux LP, Bengio Y, Tang J. RNNLogic: Learning Logic Rules for Reasoning on Knowledge Graphs. InInternational Conference on Learning Representations. [4] Ru, Dongyu, et al. "Learning Logic Rules for Document-Level Relation Extraction." Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021.

Weakness 2: Distinction between Neuro-symbolic Learning and Causal and Temporal Reasoning: The differences between neuro-symbolic learning you've pointed out and causal reasoning, temporal reasoning remain unclear. Clarifying how neuro-symbolic learning diverges from or intersects with causal and temporal reasoning approaches could enhance understanding of its unique capabilities and limitations.

Response to Weakness 2: We have revised the introduction and added a framework overview to clearly define the reasoning tasks our Neuro-symbolic Learning (NSL) method is designed to address. These revisions aim to enhance the reader's understanding of the scope and applicability of our approach.

It is important to emphasize that causal reasoning and temporal reasoning are fundamentally distinct from NSL and fall outside the scope of related work for this paper. Instead, our work is closely aligned with logical reasoning, which we have elaborated on in the related work section, with key updates highlighted in red.

Causal reasoning involves identifying and analyzing cause-and-effect relationships between events or variables. For example, it is often applied to identify risk factors for diseases by determining what causes specific health conditions.

Temporal reasoning, in contrast, deals with the timing, sequence, and dependencies of events. It is widely applied in scenarios such as planning, scheduling, and analyzing temporal knowledge graphs.

Neuro-symbolic learning, on the other hand, focuses on representing and reasoning with logical symbolic structures within a continuous, probabilistic, and differentiable framework.

The unique contribution of our work lies in going beyond traditional logical reasoning by incorporating contextual information into a distributional framework. This allows our NSL method to handle more complex scenarios where reasoning depends not only on long dependency chains but also on diverse and dynamic contextual information.

Weakness 3: Lack of Detailed Description for Training and Optimization Settings: The paper does not sufficiently detail the settings for training and optimization, which are crucial for replicating the results and understanding the model's performance under specific conditions.

Response to Weakness 3: We have added a model optimization setup with section 4.2 with red mark to describe the training and optimization settings.

Thank you for your helpful comments and questions.

Weakness 1: In general, the experiments of the paper are relatively weak. Firstly, there is a limited comparison of the proposed algorithm with other notable approaches such as DeepProblog [1] and Semantic Loss [2]. Secondly, despite the authors conducting experiments on numerous datasets, these datasets appear to be predominantly toy and simplistic.

Response to Weakness 1:

Our work focuses primarily on examining whether different types of representational spaces—fuzzy rule-based spaces, graphical probabilistic model-based spaces, and distribution-based spaces—can be inter-mapped and unified within a continuous probabilistic framework. The core objective of our research lies in exploring and formalizing the transitions within continuous representations (continuous <-> continuous).

In contrast, the scope of DeepProbLog [1] is fundamentally different, as it bridges discrete predicate logic to continuous representations by integrating neural network-based predicates (discrete -> continuous). Similarly, the scope of Semantic Loss [2] focuses on transitioning from continuous representations (e.g., neural network forward passes) to discrete label outputs for classification by employing constraint loss (continuous -> discrete).

Given the divergence in scope and objectives, we chose not to include these approaches in our experimental comparisons, as they do not align directly with the core focus of our work.

Questions 1: The author uses a bilevel optimization algorithm, can further elaborate its efficiency?

Response to Questions 1:

We acknowledge the reviewer's concerns regarding the efficiency of our bilevel optimization algorithm, particularly regarding the potential reliance on second-order information. In our implementation, we primarily employ approximate first-order optimization methods to address the bilevel optimization problem, ensuring computational efficiency. Specifically, we utilize widely adopted iterative gradient-based updating in [1] ("BOME! Bilevel Optimization Made Easy: A Simple First-Order Approach"), to implement approximate first-order updates effectively.

Questions 2: The algorithm for solving the bilevel optimization does not seem to obtain the optimal solution. Can its convergence be proved?

Response to Question 2:

This work is not a theoretical paper focused on proving convergence, and we do not make any formal claims about convergence guarantees. Instead, we rely on mature Adam optimizer to solve the subproblems with a reasonable level of accuracy. Furthermore, the algorithm is executed for a sufficiently large number of iterations, and we monitor the stabilization of iterates and variables as a practical indicator of convergence.

We respectfully request reconsideration of our score.