Learning Multiplex Embeddings on Text-rich Networks with One Text Encoder

Thank you so much for your thoughtful review!

Regarding your questions:

1. No standard deviation. Thank you for the suggestion! We conduct a two-sample independent t-test to compare our method with the best baseline (in each column). Our method outperforms the best baseline method significantly with the p-value of that t-test less than 0.05. We will clarify this in our revision.

2. Insufficient baselines. Thanks for pointing out the two baselines. We agree with the reviewer that it would be interesting to compare these methods, so we add the results on the Mathematics network and Sports network below (also added to Appendix A.12):

Mathematics

Sports

From the results, we can find that our method outperforms LinkBERT and GraphFormers consistently. The main reason is that these two baseline methods only generate one embedding for each text unit, expecting that all types of relations between texts can be captured by these single-view embeddings, which do not always hold in real-world scenarios. GraphFormers adopts the same graph propagation and aggregation module for edges of different semantics, which results in degraded performance. That being said, we agree with the reviewer that both GraphFormers and LinkBERT are related studies, we have cited these papers in our revision.

3. Math notations: m & s. Thanks for pointing it out. $m$ refers to the number of relation prior tokens used in relation-conditioned node embedding learning, and $s$ refers to the number of prior tokens used in the downstream task with learning to select. In our experiments, we find the model quite robust to $m$ and $s$. Both $m$ and $s$ are set as 5.

4. Handle unseen relations during testing. We have two ways to conduct inference for an unseen relation during the testing phase: 1) If we do not have any labeled data for the unseen relation, we can use direct inference with an evident source relation (proposed in Sec 4.2). A straightforward way is to select the source relation which has the closest semantics to the unseen relation. 2) If we have some labeled data, we can let METERN learn to select the source relations for the unseen relation inference (proposed in Sec 4.2).

Thank you so much for your thoughtful review!

Regarding your questions:

1. More detailed related works. We thank the reviewer for the comment. We have added the most relevant paper to our current related work section, but we are happy to add more related works to provide more context for readers. The newly added related works on “multi-task learning” and “learning on graphs for real-world applications” can be found in Appendix A.9. We also appreciate further suggestions for related work.

2. Computational complexity. We conduct the theoretical complexity analysis in Section 4.3 and the empirical complexity analysis in Section 5.6 (Table 6). From both the theoretical analysis and empirical results, we can find that: the time complexity and memory complexity of training METERN are on par with those of training baseline text encoders.

3. Real-world use cases and examples. The METERN model can be deployed to real-world academic paper systems (e.g., Google Scholar) and e-commerce systems (e.g., Amazon). For the academic paper systems, we can use only one model (METERN) to conduct paper analysis from multiple perspectives (semantics, topic, authorship, etc) and achieve paper recommendation, venue recommendation, collaborator recommendation, and paper analysis. For the e-commerce systems, we can deploy METERN to analyze items from different aspects (category, brand, price, etc) and conduct item recommendations, brand recommendations, and item retrieval. Potential real-world scenarios also include the literature domain and legal domain where texts are associated with rich structure information from different aspects. A more detailed discussion of real-world use cases can be found in Section A.9 in the Appendix.

Thank you so much for your thoughtful review!

Regarding your questions:

1. Novelty. Focus. We would like to argue that existing works [1] mainly focus on efficient model training, which is different from the focus of our work, which is to explore a new task of learning relation-conditioned document representations (i.e., multiplex representations on text-rich networks). Techniques. We propose a simple and effective multi-task learning method tailored for the multiplex nature of text-rich networks. The learned relation prior tokens capture the semantics of relations. These tokens can be flexibly used in the inference process either through manual selection, facilitating a plug-and-play functionality, or serving as prototype representations upon which mixup weights are automatically learned to quickly adapt to potentially intricate and unseen relations. Applications. The learned multiplex representations not only benefit relation prediction (link prediction corresponding to a specific relation) but also help various downstream tasks (shown in Tables 3,4, and 5) on the text-rich networks. We propose METERN to unify all task-solving with one model on multiplex text-rich networks.

2. GNN baselines. Thanks for pointing out the two baselines. We agree that it would be interesting to compare with a multiplex GNN equipped with pretrained text embeddings (MPNet-v2 is used here. We also tried BERT, and found that MPNet-v2 has better performance). We show the results on the Mathematics network and Sports network below (also added to Appendix A.10):

Mathematics

Sports

From the results, we can find that the performance of multiplex GNNs with pretrained text embeddings is consistently better than the performance of multiplex GNNs with bag-of-words embeddings. However, our method still outperforms both baselines by large margins.

3. Text encoder. We compare METERN with three types of baselines: 1) large-scale corpora finetuned text encoders, 2) multiplex graph neural networks, and 3) multi-relation learning language models. They have different philosophies to capture the multiplex relation semantics. The first category of methods is finetuned on very large text corpora, generating single embeddings for documents, and presuming one single embedding can capture the multiplex relation and have good generalization ability. The second category of methods learns multiplex node representations from a graph structure perspective. We show the result of this category with pretrained text embeddings in our previous answer. The third category of methods tries to capture different relation semantics with one text encoder and all these methods (including ours) adopt BERT-base as the initial text encoder for a fair comparison.

4. Natural language description of relation + LM baseline. Thanks for raising this baseline. We implement this baseline by adding a natural language description of the relation before encoding and using the large LM OpenAI-ada-002 (the best OpenAI embedding method) as the backbone text encoder. The results are shown below (added to Appendix A.11):

Mathematics

Sports

From the results, we can find that our method outperforms this baseline by a large margin, which demonstrates that the relation embeddings learned by our method to represent relations are better than directly prompting large LMs via natural language descriptions.

[1] Prefix-Tuning: Optimizing Continuous Prompts for Generation. ACL 2021.

Thank you so much for your thoughtful and detailed review!

Regarding your questions:

1. Concept drift. The concept drift does exist across different relations. As shown in Figure 6 in the Appendix, the raw data distributions of text pairs in different relations are quite different. The raw data distribution shift will result in the performance changes shown in Figure 7. In our methods, we use relation embeddings to capture the specific knowledge for different relation distributions and learn the shared knowledge across relations with the text encoder.

2. Number of relation types and prior tokens. The number of relation types is predetermined by the network of interest (e.g., for e-commerce item networks, we can have four relation types: co-purchase, co-view, bought-together, and co-brand, according to user behavior and item features). The number of relation prior tokens is a hyperparameter in our model. In our experiments, we find the model quite robust to the number of prior tokens and we set it as 5.

3. The span of associated texts. The span of node-associated texts is a flexible choice in multiplex text-rich networks. Our method proposes a generic framework to learn multiplex embeddings and is not limited by the span of text. In our experiments, we use text spans that can be encoded by pretrained language models (less than PLM max length). For papers in academic networks, we use their title/abstract; For items in e-commerce networks, we use their title/descriptions. We can also choose other types of texts (e.g., content for paper) and we leave it for future study.

4. Relationship between Transformers and METERN? This is a great point. We would like to claim the difference between Transformers and METERN from three perspectives: 1) Node (token v. document). If we treat Transformers as a special GNN, each node in the graph will be a token and the framework will finally output representations for tokens. However, in METERN, each node corresponds to a document where the model outputs are document representations. 2) Graph (token complete graph v. real-world semantic graph). If we treat Transformers as a special GNN, the graph for Transformers will be a complete graph between token nodes, while in our case, the networks are real-world graphs where the edges contain semantic information. 3) Condition (context v. relation). If we treat Transformers as a special GNN, the framework will calculate the context-conditioned token (node) representations, while in our framework, we calculate the relation-conditioned document (node) representations.

5. The example of “llama”. Thanks for the comments. The example of “llama” in the Introduction is supposed to claim the weakness of multiplex GNN methods, which represent the texts associated with each node as bag-of-words or context-free embeddings and are not enough to characterize the contextualized text semantics.

6. How to handle overlapping semantics across relations. Thanks for this great comment. 1) During training, the relation prior embeddings will capture the semantics of different relations. If two relations have overlapping semantics, it will be reflected in their relation prior embeddings (similar prior embeddings). 2) During inference, the “learn-to-select” mechanism will select related source relations for the downstream tasks. If two source relations have overlapping semantics, the “learn-to-select” mechanism will automatically learn appropriate weights by accounting for their semantic similarity.

7. Scalability of multiplex representation learning. 1) Training multiplex embeddings: Our method shares similar training time and memory complexity with text encoder baselines (Table 6). 2) Inference for downstream tasks: In practice, we do not need to save the multiplex embeddings for all documents offline. Instead, we only need to save the embeddings of the source relation which are useful for the downstream tasks. In addition, we can save the METERN parameters and conduct online inference (with our proposed two inference mechanisms) for downstream tasks.

8. How does “learn-to-select” work for novel relations? 1) If we have labeled data for this novel source relation, we can retrain the model for multiplex representation learning. Then we will obtain the relation prior embeddings for it for “learn-to-select” inference. 2) If we do not have labeled data for the novel relation but the relation is similar to one or several of our source relations, we can use the prior embeddings corresponding to the similar source relations for this novel relation for “learn-to-select” inference. 3) If we do not have labeled data for the novel relation and the novel relation is fully out of distribution compared with the source relations, there is essentially no signal in the available data to capture the semantics of this new novel relation and it should not be used in “learn-to-select” inference to guarantee performance.

Thank you so much for pointing out this interesting paper! I think the idea of partitioning sentence embeddings into multiple sub-embeddings is great. We will definitely look into it and we believe we can take inspiration from your work.