EX-Graph: A Pioneering Dataset Bridging Ethereum and X

Dear Reviewer hHyG,

We are profoundly grateful for your comprehensive review and the acknowledgment of our work's contributions. Your appreciation of our dataset, ETGraph, which innovatively links Ethereum addresses to Twitter accounts, is highly valued. The effort to construct this novel and complex heterogeneous graph, as well as its anticipated impact on blockchain research, has been a significant part of our endeavor. We are particularly gratified that you recognized the transparency and reproducibility of our work, essential qualities we strive to maintain. Additionally, your observation of the improved performance across various GNN models using our dataset, as evidenced by enhanced metrics like ROC, AUC, precision, recall, and F1 scores, affirms the practical utility and effectiveness of our research. Your insightful feedback is not only a validation of our current efforts but also a strong encouragement for our future endeavors. Regarding the weaknesses you have pointed out, we will address each of them in detail below to provide further clarification and insight.

1. The paper is not self-contained and a lot of things need to be assessed through the external links to the dataset or through the appendix. I am not very familiar with dataset papers, so maybe this is generally so.

   Thank you for your feedback regarding the self-contained nature of our paper. We understand your concern about the necessity to refer to external links and appendices for complete details. It is indeed common in dataset papers, especially those involving complex and extensive datasets like ours, to supplement the main text with additional resources. For example, [Huang et al., 2023] propose a Temporal Graph Benchmark for Machine Learning on Temporal Graphs, moving all experiment settings to the appendix and use an external website to contain many details. This approach helps maintain the clarity and conciseness of the paper while ensuring that detailed and technical aspects are readily accessible to those who seek more in-depth information.

   Also, we acknowledge your concerns and have refined our paper by incorporating some key technical details, previously in the appendix into the main text. Notably, we have added an explanation for choosing BERT, now highlighted in blue in Section 3.3 of the revised paper.

2. For this reason, I cannot comment on the ethical considerations about releasing such a dataset.

   We acknowledge and take seriously your concerns regarding the ethical implications associated with the release of our dataset. To address these concerns, we have detailed our ethical considerations and guidelines in Appendix D, focusing on the data collection procedures from OpenSea and Twitter. Throughout the development of our work, we have conscientiously adhered to Twitter's specific guidelines, notably those pertaining to off-Twitter matching and the redistribution of content for non-commercial research purposes. Recognizing the sensitive nature of personal data usage, we have implemented rigorous measures to ensure the utmost protection of user privacy. This includes the thorough anonymization of any data that could potentially be traced back to individual Twitter users, ensuring that our dataset is utilized exclusively for research objectives.

   We acknowledge the importance of addressing potential misuse of our model, such as for intrusive surveillance. Our primary goal is to explore the synergy between on-chain and off-chain data for enhancing network analytics, strictly within an academic context. Our model is designed with the intention of analyzing Ethereum transaction patterns and protecting the interests of genuine users. We want to make it clear that our model does not support, nor is it designed for, any form of widespread surveillance.

   Regarding the IRB review process, our study relies exclusively on publicly available and anonymized data, thus ensuring individual privacy and eliminating the risk of identifying specific entities. After consultation with the chair of our institution's IRB, we have been assured that our research complies with the necessary ethical guidelines and qualifies for exemption. This exemption is based on the criteria that the information we gathered cannot be used to ascertain the identities of subjects, either directly or indirectly (source: IRB Northwestern University).

7. Can you please discuss the weaknesses above?

   Thank you for prompting a discussion on the weaknesses identified in our study. We have addressed each of the weaknesses mentioned earlier in our responses.

8. Is overs-smoothing a relevant problem in the evaulations and if yes, have you taken steps to address it?

   Thank you for your question regarding over-smoothing in our evaluations. While over-smoothing is an acknowledged concern in deeper Graph Neural Network (GNN) architectures, it was not a primary focus of our current research. Our study primarily revolves around the development and utilization of a novel dataset linking Ethereum blockchain data with Twitter social network data, and exploring the insights this combined data can provide.

   Though it is not our focus, we conduct some experiments or analysis on employing 2 layers, 3 layers, 4 layers, and 5 layers in GNN models to conduct the Ethereum link prediction with and without Twitter features. We choose GraphSage as our backbone. The experiment results are attached below.

   Experimental Results: GraphSage Model Layer Comparison for Ethereum Link Prediction

   This table presents the results from experiments investigating the impact of model depth on over-smoothing in GraphSage models. We compare the performance of two, three, four, and five-layer models in conducting Ethereum link prediction, both with and without incorporating Twitter features.

   Number of Layers	Feature Set	AUC-ROC	Precision	Recall	F1 Score
   Two Layers	Without Twitter Features	0.82±0.01	0.79±0.02	0.71±0.02	0.75±0.01
   	With Twitter Features	0.86±0.01	0.82±0.01	0.72±0.03	0.76±0.01
   Three Layers	Without Twitter Features	0.84±0.00	0.77±0.01	0.77±0.01	0.78±0.00
   	With Twitter Features	0.86±0.00	0.81±0.01	0.79±0.01	0.80±0.00
   Four Layers	Without Twitter Features	0.76±0.02	0.69±0.03	0.71±0.01	0.70±0.02
   	With Twitter Features	0.75±0.03	0.70±0.03	0.67±0.01	0.69±0.01
   Five Layers	Without Twitter Features	0.71±0.02	0.65±0.03	0.67±0.03	0.66±0.03
   	With Twitter Features	0.72±0.01	0.71±0.03	0.53±0.01	0.61±0.01

   Our experimental results indicate a decline in prediction performance when the number of layers in the model exceeds three. This finding underscores the issue of oversmoothing, particularly evident from the fourth layer onwards in this dataset.

   In conclusion, increasing the number of layers beyond three introduces oversmoothing challenges. We would like to reiterate that this paper primarily focuses on introducing a pioneering dataset that bridges Ethereum and Twitter. Following your suggestion, we plan to delve deeper into the oversmoothing issue in future research using this dataset.

9. Figure 2: In the Feature Extraction column, we read in the first box "Degree"-> Structural Features. So, here, you mean that look at the degree of the node or that "Degree" is some kind of network like BERT and Deepwalk in the next two boxes of that column? I found this slightly confusing. Also, which structural features are extracted here since ETH addresses don't contain any further information.

   Thank you for your question regarding Figure 2's Feature Extraction column. In this context, the term 'Degree' specifically denotes the degree of a node within the network, rather than referring to a network type like BERT or DeepWalk. For Ethereum addresses, we have extracted a set of 8-dimensional structural features including: degree, in-degree, out-degree, neighbors, in-neighbors, out-neighbors, maximum transaction numbers with neighbors, and average neighbor degree. These features are comprehensively detailed in the Section 4.1.2 and Appendix C.1 of our paper, as space limitations in the figure restricted us from including all the specifics. We hope this clarification resolves your confusion and provides a clearer understanding of our feature extraction methodology.

Dear Reviewer hHyG,

We are sincerely thankful for your decision to increase the score of our submission, which greatly affirms the value of our work!

And thank you for your thoughtful and constructive review of our paper. We are gratified to hear that our responses and revisions have effectively addressed your concerns, particularly regarding the ethical considerations of sharing our dataset. Your confidence in removing the flag related to dataset concerns is immensely appreciated.

We also value your appreciation of the additional ablation studies included in our rebuttal. Your suggestion to possibly include these in the appendix is intriguing and something we will thoughtfully consider. It's encouraging to know that these supplementary studies, while not the main focus, have contributed positively to the overall quality of our work.

Your acknowledgment of the readability and coherence of the revised version is especially heartening, as we endeavored to make the paper accessible and clear.

Your expressed confidence in the utility of our dataset for blockchain researchers, coupled with your endorsement for the paper's acceptance at ICLR, is profoundly encouraging. We are thrilled at the opportunity to contribute to the interdisciplinary nexus of blockchain and machine learning research, and we eagerly anticipate the impact our work will have in this vibrant and evolving field.

Once again, thank you for your insightful feedback and support. Your review has been instrumental in refining our paper and ensuring its relevance and clarity.

Warm regards,

All authors

4. For task 3 is to predict whether one Ethereum address is connected to one Twitter account. In Appendix K, the authors describe "For Twitter accounts that do not have a matched Ethereum address, ... why are there Twitter accounts with no Ethereum addresses? If the negative samples are ... then does the input contain bias already?

   Thank you for your observations on Task 3 of our study. We appreciate this opportunity to clarify our methodology:

   Objective of Task 3: Our objective in Task 3 is to accurately determine if a specific Ethereum address (A) is connected to its real matching Twitter account (B). All the edges we analyze are actual matching links between Ethereum and Twitter accounts we have collected.

   Unmatched Twitter Accounts: We try our best to collect the graph as large as possible to collect more informative data from both Twitter and Ethereum, and collect matching link information as much as possible. Then, on both of them, we conduct GNN to calculate node representations based on our collected sufficient data. Though many Twitter accounts do not have matching link information, they are important because they contribute to the node representation calculation by utilizing the GNN to facilitate the message passing between Twitter nodes, in order to enhance the prediction of matching links.

   For Twitter accounts that do not have a corresponding Ethereum address, our methodology involves appending an eight-dimensional zero-vector to their respective feature sets. This procedure is vital for preserving the dimensional uniformity among all nodes within the most extensive connected Twitter graph, which is a critical factor for the optimal performance of GNN. It is important to highlight that, although these zero-vector features do not represent active attributes, they play a crucial role in upholding the structural integrity necessary for GNN analysis.

   Negative Sample Selection: In our study, we carefully curated negative samples comprising hypothetical, non-existent connections between Ethereum addresses and Twitter accounts. This methodology was intentionally adopted to enhance our model's ability to discern between genuine and artificial connections, thereby improving its overall predictive precision. It's important to note that these negative samples were meticulously selected from the pool of nodes involved in actual matching links within our dataset. To address concerns about potential biases, especially between matching and non-matching Twitter accounts, we consciously avoided choosing Twitter nodes that do not have corresponding matches in Ethereum for our negative samples. This decision aligns with the second scenario you outlined, ensuring that our experimental design remains balanced and free from unintended biases.

   Furthermore, in response to your concerns, we have conducted a supplementary experiment to investigate whether selecting non-matching Twitter accounts introduces bias in our analysis. The experiment results are attached below.

3. For task 1, the authors only use matched addresses. Since GNN can pass node features via message-passing mechanism, the semantic information is able to spread around. Whether does the semantic information benefit or harm the prediction of non-matched addresses?

   We greatly appreciate your insightful query regarding the use of matched addresses in Task 1 and the potential effect of semantic information on the prediction of non-matched addresses! In response, we conducted an additional experiment focusing on Ethereum link prediction among non-matched nodes using multiple GNN models, where the training, validation, and test sets were devoid of any matching link addresses. The results are attached below:

   Results for non-matching Ethereum addresses' Link Prediction
   (Without: without Twitter semantic information; With: with Twitter semantic information)

   Method	Experimental Setting	AUC-ROC	Precision	Recall	F1 Score
   GCN	Without	0.84±0.01	0.76±0.01	0.75±0.01	0.75±0.00
   	With	0.84±0.00(↑)	0.76±0.00(↑)	0.75±0.02(↓)	0.76±0.01(↑)
   GAT	Without	0.78±0.02	0.72±0.01	0.68±0.03	0.70±0.02
   	With	0.85±0.01(↑)	0.76±0.00(↑)	0.79±0.02(↑)	0.77±0.01(↑)
   APPNP	Without	0.84±0.01	0.80±0.00	0.70±0.01	0.74±0.00
   	With	0.87±0.00(↑)	0.79±0.00(↓)	0.77±0.00(↑)	0.78±0.00(↑)
   Cluster-GCN	Without	0.88±0.00	0.81±0.00	0.78±0.01	0.80±0.01
   	With	0.90±0.00(↑)	0.82±0.00(↑)	0.82±0.01(↑)	0.82±0.00(↑)
   TAGCN	Without	0.84±0.00	0.81±0.00	0.71±0.00	0.76±0.00
   	With	0.85±0.00(↑)	0.80±0.00(↓)	0.74±0.01(↑)	0.77±0.00(↑)

   Note: '↑' indicates an increase, '↓' indicates a decrease.

   The results of this experiment indicate that in most cases, the semantic information indeed benefits the prediction of non-matched addresses. This outcome not only corroborates the capability of GNNs to disseminate node features through the message-passing mechanism but also underscores the utility of our dataset in enhancing Ethereum link predictions, even in the absence of direct links to Twitter. Your feedback has been instrumental in guiding this valuable extension of our research. We have added this experiment to our revised Appendix I.4, highlighting its subtitle in blue color. Thank you again for your insight!

Dear Reviewer AoG7,

We express our sincere gratitude for your thorough and constructive review of our paper. Your recognition of the novelty in linking the Twitter social network with Ethereum networks, and the clear presentation of our dataset statistics, is greatly appreciated. We are also thankful for your acknowledgment of our comprehensive experimental approach, demonstrating the enhanced performance of various GNN methods with the integration of Twitter features. Your positive feedback on the soundness, presentation, and contribution of our work is encouraging and validates the efforts we have put into this research. In response to the weaknesses you have identified, we offer detailed, point-by-point explanations below.

1. Though the title says ‘bridging Ethereum and Twitter’, the proposed dataset is focused on NFT transactions and related Twitter accounts. The author should either prove that with this dataset, GNN would have the ability to find other kinds of matching links (i.e., phish-hack EOA nodes and corresponding Twitter accounts), or modify the title to be more precise.

   Thank you for your thoughtful feedback on our paper titled “Bridging Ethereum and Twitter.” We understand your concerns about the focus on NFT transactions and corresponding Twitter accounts. This title was chosen to showcase our pioneering research of the integration between Ethereum and Twitter data, considering that the NFT data was extracted Ethereum transactions. Although we recognize the benefits of a more specific title, the ICLR 2024 guidelines limit modifications to the title and abstract during the rebuttal phase. Should our paper be accepted, we will revise the title to “ETGraph: A Pioneering Dataset Bridging Ethereum NFTs and Twitter” for the camera-ready version, in line with ICLR’s policy for minor adjustments. We appreciate your valuable suggestion!

2. In Section 3.3, the authors claim, ‘we obtain embeddings for all Twitter accounts using the DeepWalk algorithm in the Twitter graph’. Figure 2 also shows structural features given by deep walk. However, as stated in Appendix, only the handcrafted 8 features from C.1 are used.

   We are grateful for your attention to detail in highlighting the discrepancy between Section 3.3 and the Appendix C.1 of our paper. You are correct in noting that we initially neglected to include the details regarding the use of the DeepWalk algorithm for obtaining Twitter features in the Appendix C.1. We have addressed this oversight and updated Appendix C.1 to correctly depict the use of DeepWalk, now highlighted in blue, ensuring consistency with the methodology outlined in Section 3.3 and Figure 2. Thank you for bringing this to our attention, allowing us to improve the clarity and consistency of our paper!

Comparative Results for Task 3: Ethereum-Twitter Matching Link Prediction

This table showcases the findings from our experiments for Task 3, centered on predicting the links between Ethereum addresses and Twitter accounts. In our comparative analysis, we evaluated the performance of two distinct setups: our original configuration, which employs matching Twitter accounts, and an alternative approach where we utilized non-matching Twitter accounts as negative samples. This comparison aims to provide a clearer understanding of the impact of negative sample selection on the accuracy and reliability of our predictions.

Note: '↑' indicates an increase, '↓' indicates a decrease, and '-' indicates no significant change in performance.

The results of our experiment offer a clear indication that selecting non-matching Twitter accounts as negative samples significantly influences prediction accuracy, leading to near-perfect scores close to 100%. This high level of accuracy can be attributed to the fact that Twitter accounts without corresponding Ethereum addresses lack distinct Twitter profiles in our dataset. Instead, we use an eight-dimensional zero vector as a placeholder for them. Such uniformity in features makes these accounts easily identifiable during the training of Graph Neural Networks (GNNs), leading to an unusually high prediction accuracy, which is a bias as you mentioned. This finding corroborates your concern regarding the introduction of bias in our analysis when non-matching Twitter accounts are used as negative samples.

Last but not least, we sincerely thank you for your thoughtful and detailed review of our work. Your insightful feedback has been instrumental in refining our approach and methodology, particularly in the nuanced aspects of handling unmatched Twitter accounts and the selection of negative samples for our GNN analysis. Your questions and observations have not only helped in clarifying these key areas but have also contributed significantly to enhancing the overall quality and clarity of our research. We are truly grateful for the time and effort you have dedicated to reviewing our paper.

Dear Reviewer SzMQ,

We are sincerely grateful for your recognition of our contribution, particularly the introduction of our novel graph dataset, ETGraph. This dataset, which uniquely combines Ethereum and Twitter data, holds promising potential for impactful applications in both the blockchain and machine learning communities. In response to the weaknesses you have highlighted, we provide detailed explanations below.

1. As the paper suggests a new dataset, it is acknowledged that the technical novelty is not strong enough.

   Thank you for bringing up this point. We acknowledge your concerns about the technical novelty of our work. Nonetheless, it's important to emphasize that the development of our dataset transcends simple data aggregation. Our process entailed intricate data collection from Ethereum and Twitter, followed by sophisticated integration and preprocessing tasks, such as graph construction and feature engineering. These elements collectively enhance the technical complexity of our project, setting it apart in the landscape of machine learning research.

2. But as the authors provide a webpage and github page to easily use the dataset, it may contribute to the ML community.

   Thank you for acknowledging our contribution to the ML community. Our decision to provide easy access to the dataset through a well-organized website and GitHub repository is a conscious effort to support and stimulate further research. We believe that this accessibility is a vital contribution to the machine learning community, facilitating broader use and exploration of ETGraph.

Once again, we extend our gratitude for your thoughtful review and the valuable insights you have provided. Your feedback has not only strengthened our research but also serves as a source of motivation as we continue to pursue innovative endeavors within the machine learning community.

Q4: As an example, the set of features degree (3 features) and neighbor (3 features) seem to be highly linearly correlated (as also suggested by their statistics in Table 3). Keeping only of them, i.e., one from the "degree" family, and one from the "neighbor" family, would potentially improve the performance of the networks. (Reviewer #hHyG)

We have conducted additional Ethereum link prediction to evaluate the impact of retaining only one feature from each of these categories – one from the 'degree' family and one from the 'neighbor' family.

The results from our experiments provided insightful findings. When we limited the features to just one from the degree family and one from the neighbor family, we generally observed a decrease in prediction performance across the two experimental settings, particularly in the predictions involving Twitter features. This trend highlights the significance of utilizing a comprehensive set of features in our model to effectively capture the intricate details and nuances of our dataset.

Q5: Is overs-smoothing a relevant problem in the evaulations and if yes, have you taken steps to address it? (Reviewer #hHyG)

Though it is not our focus, we conduct some experiments or analysis on employing 2 layers, 3 layers, 4 layers, and 5 layers in GNN models to conduct the Ethereum link prediction with and without Twitter features. We choose GraphSage as our backbone.

Our experimental results indicate a decline in prediction performance when the number of layers in the model exceeds three. This finding underscores the issue of oversmoothing, particularly evident from the fourth layer onwards in this dataset.