Multiple Heads are Better than One: Mixture of Modality Knowledge Experts for Entity Representation Learning

Dear Reviewer:

Thank you for your insightful feedback. Below are our responses to each of your comments and questions. Hope our response finds you well and improves your impression on our paper.

About the design in Section 4.1

Yes, $W_{m, i}$are learnable projection layers (nn.Linear in PyTorch with bias). They are in the shape of (hidden state dimension, modality feature dimension), aiming to first project the raw modality features into a hidden space and obtain the multi-pespective embeddings $Vm,i$of each entity. Since W is not only a linear layer but also contains bias, we use this representation.

About Temperature in Equation (2)

This temperature is a relation-wise parameter. For each relation, we set one learnable parameter as the temperature. It is used to change the experts's weights when facing different relation prediction tasks. As we introduced in Figure 1 and Section 3, multi-modal knowledge graph completion (MMKGC) is a triple-level task. For a given head entity h, when we predict under different relation r, the answer would be different. Therefore, we expect that the MMKGC model can learn different entity representations under different relations for better prediction. Therefore we design such a temperature that introduces the information of relations when learning entity representations. Compared to traditional methods that introduce relationship information only at the time of final prediction, our approach can be analogized as an early fusion of features.

About Equation (3)

Sorry for this typo and we have fixed it in the revised version of our paper.

About the joint modality

The issue of the use of Joint Score is specifically explored in the ablation study, as you can see in Table 3 of Section 5.5 we explored the performance of each modality individually for prediction, and from the results of this experiment, we can also see that adding all the modalities together will have the effect of From the results of this experiment, we can also see that adding all modalities together will result in a better model and that a model trained using any of the modalities alone (including joint modality) will not be as good as a full model.

The setting of expert number

This parameter is related to the design of Section 4.1. Our model has K experts in the ReMoKE module to obtain K different multi-perspective embeddings of each entity. We further make an exploration about the hyper-parameter K in Section 5.5. Thanks for pointing this out, we've added this section to the paper's implementation details.

The difference between MKG-W and MKG-Y

MKG-W and MKG-Y are two different datasets proposed in MMRNS. Both of them are constructed based on the OpenEA project. However, they are two different MMKGs because their data sources are different. The modality information is collected by search engines. MKG-W is based on Wikidata and MKG-Y is based on YAGO. Both of the data sources are famous open-source knowledge graphs.

These are our responses to your comments and questions, and we hope that our responses will answer your concerns and improve your impression of our paper. We hope that you can improve the score of your decision, which will mean a lot to us.

About visualization (w/ Q3)

The phenomenon you mention is related to the design of our model. The reason has two main parts, within each modality expert, we form a differentiation within each modality through MoE design, so that each expert learns different contents, and between modalities, we again highlight the weight of the key modalities through the dynamic modal feature fusion, so that the synergistic effect of the two bootstrapping makes the model internal for the entity's modal weights appear distributed as shown in the visualization results.

And we plot the adaptive weight distribution within a single modality and across multiple modalities in the visualization results. We can see that not only do the experts within a modality form a differentiation, but the weights of the modalities to each other are also significantly different in different RELATION contexts.

These are our responses to your comments and questions, and we hope that our responses will answer your concerns and improve your impression of our paper. We hope that you can improve the score of your review, which will mean a lot to us.

The design of expert network (also w/ Q1)

The expert networks treat the raw modality embeddings as the inputs and produce relation-aware entity embeddings as outputs. In this process, we use project layers (W) and mixture-of-experts architecture to capture the representative information in the given embeddings. Note that it is a common setting for embedding-based MMKGC methods, which means that all the MMKGC baselines we used in the experiments follow such a setting and use the raw modality information as inputs with further process pipelines. From the experiment results, we can observe that MoMoK does well in leveraging the existing information in the raw embeddings compared with baseline methods.

But your concern is still valuable, how to fully unleash the power of raw modality data (images and texts) is another significant topic in multi-modal representation learning, which will be our future direction.

To solve the problem you mentioned in Question 1, our MoMoK framework employs multiple modality experts in each modality, which can learn some important and key features from diverse perspectives. We design tunable Gaussian noise and relation guidance to enhance this module. The tunable Gaussian noise is a classic design in MoE [3] which can guide sparsification learning for experts in MoE. The relation guidance is our new design considering the realities of the MMKG scenario.

[3] Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.

More about the CLUB loss in Equation 8 (also w/ Q2)

Equation 8 is our core design for expert information disentanglement. Note that we have set multiple experts in each modality. We expect these experts can learn intra-modality knowledge from different perspectives. Therefore, we design such a module to guide different experts to learn different aspects by considering the mutual information. Mutual information is an important concept in information theory and modern deep learning, which can measure the dependence of two variables.

Famous representation learning technologies, such as InfoNCE, are widely used to maximize mutual information. Maximizing mutual information enhances the dependence and correlation of two variables while minimizing mutual information allows two variables to be decoupled from each other. In our design, we choose to disentangle the experts. Therefore, we employ CLUB, a famous mutual information minimizing framework in our design. We provide a more detailed theoretical analysis in the Appendix to present our opinion.

More related work analysis

Due to the page limitations of the submission, we did not provide an in-depth analysis of the differences between the different MMKGC methods in the related works of the manuscript. However, we have made a preliminary delineation in the introductory section of the baselines of the Appendix. We'll refine this part in the revised submission. For this question of your concern about the difference between MoMoK and MoSE, we've organized it with this table below:

Though MoSE, IMF, and our method MoMoK are all ensemble-based MMKGC methods, MoMoK differs in many ways from the two existing methods. Compared with MoSE and IMF, MoMoK has a relation-aware multi-modal fusion module (ReMoKEs) and an intra-modality disentanglement module (ExID), which makes MoMoK outperform the baselines according to our experiments.

Note that we have revised our paper and added more related works analysis in the Appendix A.

The learning of structural embedding

In our design, the structural embeddings are learned during training. This is different from the way raw embedding is handled for image and text, where the raw embedding for these modalities is pre-extracted, whereas structural embedding is learned in training after random initialization. This is a common setting in MMKGC models.

Typos

Sorry for the two typos, we have fixed it in our revised version.

Dear Reviewer:

Thank you for your insightful feedback. We have made a response to each of your comments and questions. Hope our response finds you well and improves your impression on our paper.

Novelty of our method

In KGC prediction, relation context is something that needs to be taken into account, and the vast majority of traditional approaches are to train relation embeddings individually and apply these relation embeddings to the calculation of scores. While MoSE is an ensemble learning MMKGC method, its underlying backbone is the classic ComplEx model, and the relation context is considered when doing the ensemble. This consideration of the relation context remains at the final stage. The entity embeddings of these methods would not be changed when facing different relation contexts. The relation context only appears in the final step of these models.

However, our proposed framework, MoMoK considers the relational context in a more early stage when obtaining the entity representation. We achieve this goal by the design of relation-guided modality knowledge experts. In the subsequent score calculation, the relation context will still participate in the score calculation in the form of embeddings. In this way, we consider the relation context in both phases of the model, which also makes our entity embedding learning can change dynamically with different relation contexts during both the training and inference stages, which is one of our novelties.

More results on more datasets

We find this question valuable and therefore also complement the results of our approach on a more standard dataset. Current research on uni-modal KGC usually employs FB15K-237 and WN18RR for evaluation. Other datasets like FB15K / FB-IMG / WN18 have certain problems that result in lesser use. It is because there are some trivial patterns and inverse relations [1, 2] in these datasets that are easy to be learned by the models, making these datasets lack of generalization and challenge, leading to very high results for most of the models and making it difficult to differentiate between the capabilities of different models. According to our attempts, the WordNet series of datasets have higher-order abstractions in their corresponding image information because their entities are words, making these images less useful for MMKGC prediction. At the same time, this part of the information is also more difficult to obtain, and the download methods provided in some open-source repositories require more complicated configurations to obtain the full images.

Therefore, we present some more results on FB15K-237, shown in the following Table. The results with * are from the MoSE paper. - means the results are not reported in the given paper.

These results indicate that MoMoK still performs well on classic MMKG benchmarks like FB15K-237. Only MoSE-BI is currently slightly better. Compared with recent baselines like AdaMF, MoMoK performs better. Based on our observation of the source code, MoSE achieves this result by setting the dimension of embedding to 2000, while our method MoMoK uses 250.

Compared with MoSE, MoMoK is an end-to-end training framework, MoSE needs to learn the parameters of the ensemble once more after the model is trained, which is also a feature of our design.

[1] Convolutional 2D Knowledge Graph Embeddings

[2] Observed Versus Latent Features for Knowledge Base and Text Inference

Figure font

Sorry for this typo, we have fixed it in the revised version of our paper. You can find it in the uploaded PDF of our paper.

Detailed setting of Section 5.4

In Section 5.4, we make a further exploration of the MMKGC performance under more complex scenarios. We set three different kinds of scenarios: modality noisy scenario, modality missing scenario, and link sparse scenario. Each of these three scenarios has a different setting.

• Modality noisy scenario. Modality information for a certain ratio of entities is added with the noise sampled from a normal distribution.
• Modality missing scenario. Modality information for a certain ratio of entities is missing.
• Link sparse scenario. The triple in the training set is reduced by a certain percentage, making the data sparser.

For all of the three scenarios, we set a ratio (noisy rate, missing rate, and sparse rate) when conducting the experiments. The noise rate is set in 10%, 20%, 30%, ......, 90%. The missing rate is set from 10% to 50% with a margin of 10%. The sparse rate is set from 5% to 25% with a margin of 5%. For a given ratio, we select the same group of entities/triples and add modifications to them. For evaluation, we follow the same protocols as the main experiments and draw some line graphs to represent the MRR and Hit@10 results under the mentioned settings.

Advantage of embedding-based MMKGC

Although there are gradually more LLM-based KGC methods nowadays, these kinds of methods and the embedding-based methods we explore in this paper are two developmental routes. LLM-based methods rely entirely on textual information about entities and relations and utilize the powerful textual comprehension and generation capabilities of LLM for text-based prediction. These kinds of methods are very useful for knowledge. This type of approach is less capable of utilizing the rich graph structure information present in the graph, and also requires fine-tuning the LLM, resulting in a higher cost for its training and inference.

One of the advantages of the embedding-based approach is the high efficiency of training and inference, and the second is the more flexible utilization of modal information, which only requires the input of modal eigenvectors of the entity for prediction, and does not strictly require the presence of explicit text. In the case of modal information is partially missing or noisy, it still has better performance. Of course, the combination of embedding-based and LLM-based methods will be the future development direction, and the combination of Knowledge Graph and LLM will also be an important research direction.

Hope that our response finds you well and can improve your assessment of our paper. We would greatly appreciate it if you could consider raising your score.

These are our responses to your comments and questions, and we hope that our responses will answer your concerns and improve your impression of our paper. We hope that you can improve the score of your decision, which will mean a lot to us.

Dear Reviewer:

Thank you for your insightful feedback. We have made a response to each of your comments and questions. Hope our response finds you well and improves your impression on our paper.

More exepriment analysis

In our experiments, we present the MRR and Hit@1 results in Table 2. We now present the results for the complete set of four metrics in both datasets in the table below:

We can draw more conclusions from these added experimental results. We can find that our method MoMoK outperforms baselines on MRR/Hit@1/Hit@3 on MKG-W and Hit@1/Hit@3 on MKG-Y. We can find that our method will perform better on fine-grained ranking metrics such as Hit@1 than on the coarse-grained ranking metric Hit@10. The trend reflected from the new results is the same as DB15K and KVC16K. Since MRR is calculated as an average of the inverse of all the ranks, it will be affected by both the coarse and fine ranking capabilities of the model.

Intergration of Larger Pre-trained Multi-modal Models

In our paper, the dataset we used is more diverse, and they used several different backbone models in obtaining text and image embedding of entities. And BERT and VGG are representative examples of them. We detail the dimensions of image and text embedding used for each data in Table, while the feature dimensions are different for different datasets and the pre-trained model used is also different, which we summarize below:

We can find a rich variety of backbones used in the course of our experiments. Our approach is an in-depth extension of the traditional embedding-based approach, while the other class of KGC methods is the one you mentioned, based on large multi-modal models for fine-tuning and similarity comparison, which requires training of the large multi-modal model itself. Often, they require very much computational resources and the results of training and inference are slow.

Nowadays, the more mature multi-modal model-based MMKGC methods are MKGformer [1] and SGMPT [2]. We report the results on the FB15K-237 dataset. We present the results in the following Table. The results with * are from the original paper.

From the experimental results, we can also see that our method performs better in all aspects of the metrics compared to these methods that directly use LARGE multi-modal models, and at the same time, from the information provided in the original paper, the parameter sizes of these models are around 950M and take close to 10 hours to train. This is also low compared to the efficiency we show in Table 4, so our MoMoK does better in terms of performance and efficiency compared to such methods. Of course, this will also be our future direction.

These are our responses to your questions. We hope that our response will solve your concerns and improve your rating of our paper, and we would greatly appreciate it if you could consider improving your score.

[1] Hybrid Transformer with Multi-level Fusion for Multimodal Knowledge Graph Completion

[2] Structure Guided Multi-modal Pre-trained Transformer for Knowledge Graph Reasoning

These are our responses to your comments and questions, and we hope that our responses will answer your concerns and improve your impression of our paper. We hope that you can improve the score of your decision, which will mean a lot to us.

Dear Reviewer:

Thank you for your insightful feedback. We have made a response to each of your comments and questions. Hope our response finds you well and improves your impression on our paper.

Model parameter size analysis

We think this comment is very valuable, so we have dutifully analyzed further model parameter quantities.

• For traditional uni-modal KGC methods, their core parameters focus on the embedding of the entities and relations. Each embedding is a d-dimensional vector.
• For multi-modal KGC methods, they have additional parameters including raw modality features ands pecific multi-modal fusion modules.
• For negative sampling methods, they have additional parameters in the negative sampling module.

At the same time, the number of parameters can vary greatly from one method to another due to differences in their design. Next, we counted the number of parameters for different models on the DB15K dataset with d = 250. We used the same few methods from Table 4 to make a fairer comparison.

From this table, we can find that our method MoMoK consists of a few more parameters compared with MMKRL and OTKGE. Though recent SOTA methods like OTKGE and MMRNS have fewer parameters, they design very complex computations and processes on their modules. For example, OTKGE implements an optimal transport algorithm in its model and MMRNS designs complex negative sampling strategies. These designs can greatly affect the training efficiency and GPU memory usage. Considering the performance, efficiency, GPU memory, and parameter number, the performance progress and efficiency changes achieved by our method are acceptable.

More about Table 4

In Table 4, the GPU memory and time cost refers to the training stage. We compared our method with several recent baselines to demonstrate that our method is time and space-efficient during the training time. This comparison is also made because MMKGC models tend to have more complex designs in the training phase, such as adversarial training (MMKRL), optimal transport (OTKGE), adaptive negative sampling (MMRNS), etc., which can significantly slow down the training of the model. We demonstrate the efficiency of our approach through this experiment. The time cost refers to the training time for 1 epoch.

The loss weight \lambda

In our design, we add a weight $\lambda$to the ExID module for the CLUB loss. This loss aims to disentangle the mutual information in the modality expert networks, serving as an auxiliary loss. The current choice of parameters is an empirical conclusion that if the weights are set too large, they will affect the main training goal of the MMKGC, whereas a smaller number can give an effective guide to the model training. The nature of this loss is similar to the classical contrastive loss, which is used for the estimation of mutual information, although they have different end goals; contrastive loss aims at maximizing mutual information, while CLUB loss aims at minimizing mutual information. From this point of view we can also give an explanation that mutual information between different exerts should guarantee a certain degree of disentanglement, but outright disentanglement instead has no gain in performance.

These experimental results are consistent with those shown in the paper for MKG-W and DB15K. That is, each module contributes to the final performance. Besides, we can also do some analysis from the new experimental results on this issue of the small impact of the different modules that you mentioned. On the one hand, we conducted a fine-grained ablation study, with specialized experiments on several key designs. On the other hand, we can observe that there is also a big difference in the contribution of the same module in different datasets. This is related to the dataset itself, in addition to the fact that the foundational elements of our entire framework, such as the TuckER score itself, have better performance and subsequent designs have made significant enhancements based on this foundation.

These are our responses to your comments and questions, and we hope that our responses will answer your concerns and improve your impression of our paper. We hope that you can improve the score of your decision, which will mean a lot to us.

Compared with MoSE, MoMoK is an end-to-end training framework, MoSE needs to learn the parameters of the ensemble once more after the model is trained, which is also a feature of our design.

[1] Convolutional 2D Knowledge Graph Embeddings

[2] Observed Versus Latent Features for Knowledge Base and Text Inference

Modality Data and Encoder Details

For the modality data in each dataset, we follow the setting of their original paper and use their released multi-modal features. We present their dimensions in Table 1 and further list the pre-trained backbone used for each dataset in the following table:

This information could be also found in the original papers. Note that in the paper, we just mention BERT and VGG as an examples of the more diverse backbones that will be used in the actual dataset, and to ensure a fair comparison of the experimental results, we use the same modality features across baselines. When handling multiple images, the entity visual embedding would be an average of multiple image features.

Notation Standardization

We will continue to optimize the formula notations in our paper. In formula (1), the notation $\mathcal{E}/\{h\}$represents the entity set without h. It is not $\mathcal{E}_{h}$ . Perhaps there is some misunderstanding. We have changed this symbol to a "minus sign (-)" for easier understanding.

About the temperature

The relation-wise temperature is a learnable scalar parameter for each relation. We first initialize them randomly and they can be optimized during training. When calculating the weights in Equation (2), each relation-wise temperature is used with a sigmoid function to limit its range to (0, 1). By doing this, we can adjust the weights of each expert with guidance from relations.

Score Function Comparision

Different MMKGC baseline methods have their specific designs, and we have selected as many recent SOTA methods as possible for comparison in Table 2 of the main experiments, and also provided the performance of the original TuckER score, and from the comparisons of these results, we can find that we are in the Tucker's performance has made a significant improvement. This suggests that the improvement we achieved is based on better utilization of modal information. This is also evidenced in the ablation study.

More analysis and result about the ablation studies (weakness 7 & 8)

We found this comment so valuable that we did not put this part of the results in the initial manuscript because of typographical issues. Now we put the experimental results of MKG-Y and KVC16K in the list below and do a more in-depth analysis of the whole ablation study.

Dear Reviewer:

Thank you for your insightful feedback. We have made a response to each of your comments and questions. Hope our response finds you well and improves your impression on our paper.

More results on MKG-W and MKG-Y

In our experiments, we present the MRR and Hit@1 results in Table 2. We now present the results for the complete set of four metrics in both datasets in the table below:

These added experimental results allow us to draw further conclusions. For example, we can find that our method MoMoK outperforms baselines on MRR/Hit@1/Hit@3 on MKG-W and Hit@1/Hit@3 on MKG-Y.

We can conclude that our method will perform better on fine-grained ranking metrics such as Hit@1 than on the coarse-grained ranking metric Hit@10. The trend reflected in the new results is the same as DB15K and KVC16K.

More results on other datasets

We find this question valuable and therefore also complement the results of our approach on a more standard dataset. Current research on uni-modal KGC usually employs FB15K-237 and WN18RR for evaluation. Other datasets like FB15K / FB-IMG / WN18 have certain problems that result in lesser use. It is because there are some trivial patterns and inverse relations [1, 2] in these datasets that are easy to be learned by the models, making these datasets lack of generalization and challenge, leading to very high results for most of the models and making it difficult to differentiate between the capabilities of different models. According to our attempts, the WordNet series of datasets have higher-order abstractions in their corresponding image information because their entities are words, making these images less useful for MMKGC prediction. At the same time, this part of the information is also more difficult to obtain, and the download methods provided in some open-source repositories require more complicated configurations to obtain the full images.

Therefore, we present some more results on FB15K-237, shown in the following Table. The results with * are from the MoSE paper. - means the results are not reported in the given paper.

These results indicate that MoMoK still performs well on classic MMKG benchmarks like FB15K-237. Only MoSE-BI is currently slightly better. Compared with recent baselines like AdaMF, MoMoK performs better. Based on our observation of the source code, MoSE achieves this result by setting the dimension of embedding to 2000, while our method MoMoK uses 250.

Dear PCs, SACs, ACs, and Reviewers:

We have also uploaded the revised version of our paper after fixing the typos and adding more contents in it.

Sincerely,

Authors

Dear Reviewers:

We have now uploaded a new revision of our paper. In this revision, we add more experimental results mentioned in the review comments. We hope you can find time in your busy schedule to take a look. Your feedback is very important to us.

Sincerely,

Authors