BlockFound: Customized blockchain foundation model for anomaly detection

Reviewer's Comment: The introduction of this paper needs to sufficiently explain the motivation behind utilizing an LLM for detecting anomalous transactions in blockchain environments. ......Additional references or evidence should be provided to justify the need for anomaly detection in blockchain transactions.

Response: Thank you for your insightful comments. While we acknowledge that consensus protocol vulnerabilities have been studied extensively in the academic literature, we note that such vulnerabilities are often very challenging to exploit in real-world blockchain environments. To the best of our knowledge, there are few, if any, practical instances of attacks exploiting consensus-level vulnerabilities in deployed systems in recent years.

In contrast, most real-world attacks target logic or implementation flaws within smart contracts, which can manifest as anomalous transaction behaviors. Therefore, our study focuses on detecting anomalies at the transaction level to identify these types of vulnerabilities. By detecting such anomalies, our approach enables transaction emergency stop, which can help prevent losses when malicious behavior is detected. Prior works also highlight the importance of anomaly detection in minimizing the loss and upholding the integrity[1,2,3].

We have incorporated additional references in the revised paper to further support this focus on transaction-level anomaly detection in Section 2 in the revised paper.

[1] Blockchain Large Language Models

[2] Detecting Anomalies in Blockchain Transactions using Machine Learning Classifiers and Explainability Analysis

[3] Anomaly Detection in Blockchain Networks: A Comprehensive Survey

Reviewer's Comment: The paper needs to clearly define what is an anomalous transaction. The ground truth for such anomalies is not clear. Only with a known ground truth of anomalous transactions can we discuss whether "using the reconstruction errors as the metric for anomaly detection" is appropriate.

Response: Thank you for your question regarding the definition of anomalous transactions and the ground truth for these anomalies. In our study, we define anomalous transactions as those exhibiting behaviors that deviate significantly from typical transaction patterns, often due to logic or implementation flaws in smart contracts. Examples include transactions with abnormal method calls or sequences that differ significantly from expected patterns in benign transactions.

For the ground truth, we have collected a set of known real-world malicious transactions, verified through reputable sources, including sources like CertiK and PeckShield. These transactions serve as the basis for defining anomalies, providing a clear ground truth to evaluate the model’s effectiveness.

Using reconstruction errors as the metric for anomaly detection is appropriate here, as it captures deviations from learned benign patterns. Anomalous transactions tend to have higher reconstruction errors due to their irregularity, which aligns with our detection approach. We have further clarified this point in Section 4.1 in the revised paper.

Reviewer's Comment: There is a need for more detailed analysis on the reliability of the datasets used (types, the number of training samples, the number of testing samples, etc.). For example, the statement "our Ethereum dataset consists of 3,383 benign transactions for training, 709 benign transactions for testing, and 10 malicious transactions. The data was collected from October 2020 to April 2023." raises questions. Does this imply there were only ten malicious transactions on Ethereum from October 2020 to April 2023, or were these ten selected by the authors? If selected, what were the criteria for their selection?

Response: Thank you for your question regarding our dataset. We apologize for any confusion. The data we collected focuses on specific DeFi applications within the Ethereum network, rather than all transactions on the entire Ethereum blockchain. The ten malicious transactions included were drawn from 5 DeFi applications and represent recorded and verified attack instances within that context.

Our choice of these transactions is based on their availability from trusted sources that confirm their malicious nature. Due to the limited number of recorded and verified attacks specific to these applications, our dataset reflects the available data within this scope. We have highlighted this selection criterion in Section 5.1 and Section 6 in the revised paper.

Reviewer's Comment: The paper primarily conducts experimental evaluations on Ethereum and Solana networks, lacking validation of the model's generalization capability and adaptability.

Response: Thank you for emphasizing the importance of generalization and adaptability across blockchain platforms. We agree that validating our model on diverse platforms is valuable. In this work, we selected Ethereum and Solana as representative platforms due to their widespread usage and differing architectural characteristics, which allow us to demonstrate our model's applicability across varied blockchain types. Given the non-trivial nature of data collection for additional platforms, we have focused on these two as a starting point and will consider expanding to other platforms in future work. We have also added this point to our discussion section in the revised paper.

Reviewer's Comment: The paper highlights the model's performance but lacks transparency in decision-making. Integrating tools could improve interpretability and enhance user trust.

Response: Thank you for this valuable suggestion. We agree that interpretability is important, as it can help enhance user trust and provide greater transparency in the model’s decision-making process. In this paper, our primary focus is on establishing effective models rather than building new interpreting methods for foundation models; however, we recognize the importance of incorporating interpretability tools and have highlighted this point to our discussion section as an area for future work.

Reviewer's Comment: In handling blockchain transaction data, privacy protection is a crucial consideration. How does the article address and safeguard user privacy data?

Response: Thank you for this important question. We are committed to upholding data privacy standards. In our work, we utilize only publicly available transaction data from blockchain networks, which are fully transparent and downloadable by design. No private or permissioned blockchain data is used. Additionally, user addresses in these transactions are represented as hashed identifiers, inherently protecting user-sensitive information and making it impossible to trace back to personal identities. This ensures that our research respects and safeguards user privacy. We have highlighted this point in our discussion section in the revised paper.

Reviewer's Comment: How does fine-tuning the GPT-4 model using the API enhance its performance for the same tasks or domains?

Response: Thank you for this suggestion. We conducted experiments fine-tuning GPT-4 (version 2024-08-06) using the Ethereum dataset to learn benign transaction patterns. Our approach was inspired by BlockGPT but with modifications due to the constraints of OpenAI’s fine-tuning API. Specifically, we used the first half of each benign transaction as input and the last half as the target response, enabling the model to learn transaction prediction. The error between the predicted and actual transaction is used as the anomaly score.

Unlike BlockGPT’s token-by-token iteration approach, we opted for this method because:

1. The current OpenAI fine-tuning API supports only instruction-response style fine-tuning, not token-by-token iteration.

2. Generating $N−1$ training samples for a transaction with $N$ tokens would result in prohibitively high costs. Even with our modified approach, the fine-tuning and inference process cost approximately $950.

Our results, included in Appendix C.2 in the revised paper, show no significant improvement compared to using GPT-4o directly without fine-tuning. Possible reasons include:

1. The model might require more training iterations than our budget allowed.

2. The half-half prediction approach might be too coarse-grained to capture fine transaction details.

3. The default tokenization might make it difficult for GPT-4 to handle specific patterns, such as addresses and numbers.

Further exploration of these issues would require greater control over GPT-4 fine-tuning and tokenization, which is currently not feasible. However, we believe a more fine-grained fine-tuning approach, potentially integrating tool usage, could improve performance. We have added this as a future direction in our discussion section.

Reviewer's Comment: Some seem too subjective as no relevant cases or references were found, for example: “..these models cannot capture the long-range dependencies and complex temporal dynamics inherent in transaction data, resulting sub-optimal modeling performance...” in Section I and "...using MLM can significantly reduce the computational cost...." in Section III.B

Response: Thank you for pointing this out. We have revised the paper to include relevant references and correct the claim we made. To clarify:

1. The advantage of transformers over LSTMs in capturing long-range dependencies has been well-documented in prior studies [1, 2, 3], which we have now cited in the revised paper.

2. Regarding the computational cost of using MLM versus GPT-style models, we appreciate you bringing this to our attention. After re-evaluating our statements, we recognize that our previous claims regarding the computational cost differences between MLMs and GPT-style models were inaccurate. For the anomaly detection task, with the given sequence, we can input the full sequence into the model in a single forward pass to compute the hidden states and output logits for each position in parallel. Thus, the time complexity for both MLM and GPT-style models is $O(N^2)$. We have updated the manuscript to correct this oversight. Our primary objective is to learn meaningful representations of transaction patterns to identify anomalies effectively, rather than to generate new sequences. While GPT-style models are powerful for generative tasks, MLMs offer an effective framework for encoding input data without the necessity of learning sequence generation. This makes MLMs particularly well-suited for our anomaly detection task.

We have included the reference and the correct the claim in Section 1 of the revised paper to clarify these points. Thank you again for your valuable feedback, which has helped us improve the accuracy and clarity of our paper.

[1] Stabilizing transformers for reinforcement learning

[2] A comparison of transformer and lstm encoder decoder models for asr

[3] Transformers in Time Series: A Survey

We thank the reviewer for the careful reading and valuable suggestions. Below are our responses:

Reviewer's Comment: Although the adapted foundation model shows some performance improvement with RoPE and FlashAttention, it represents a typical data mining method that has been extensively applied to various tasks, thus limiting its originality.

Response: Thank you for the comment. We agree with the reviewer that RoPE and FlashAttention have been widely used in training foundation models. In this work, we adopt them to train the SOTA foundation model for blockchain transactions. We believe they are reasonable design choices because they help address our key challenge in blockchain transaction foundation model development: long sequence.

We would like to kindly point out that our main technical contribution is the introduction of multi-modal tokenization and a post-training detection approach. The multi-modal nature of transaction data introduces significant complexity, and our tokenization technique is specifically tailored to manage this, ensuring the model effectively interprets distinct transaction features. Furthermore, the post-training detection step allows our model to identify anomalies within a transaction context effectively.

To the best of our knowledge, our work is the first to combine advanced model construction techniques with our customized designs for training blockchain foundation models. Given our customized designs and the effectiveness of the trained model, we respectfully point out that applying RoPE and FlashAttention does not dilute our contribution.

Reviewer's Comment: In Tokenizer, all tokens share the same vocabulary, since unique hash addresses are individual tokens, how can the size of this vocabulary be controlled to make it available at all times?

Response: Thank you for your question. In our approach, we initially build the tokenizer using a large transaction corpus. For unique addresses, we rank them by frequency and retain the most common ones (e.g., the top 7,000) as unique tokens, while replacing less frequent addresses with an [OOV] token. We acknowledge that this approach has limitations; if the current data distribution diverges significantly from the distribution used to create the tokenizer, a high number of [OOV] tokens could potentially degrade performance. To mitigate this risk, we recommend that developers periodically rebuild the tokenizer, either after a specific number of transactions or when a noticeable shift in data distribution occurs. This will help maintain optimal detection performance by keeping the tokenizer aligned with current transaction patterns. Thanks for pointing this out and we have added this to our discussion in Section 6 in the revised paper.

Reviewer's Comment: FlashAttention is a key module to handle long inputs in the foundation mode. However, there has been no experimental evidence that this approach has any impact on the accuracy or complexity of the model.

Response: Thank you for raising this point. We have conducted an ablation study, which we have included in the revised paper, to evaluate the impact of FlashAttention on both training speed and memory efficiency. For Ethereum, FlashAttention reduces the running time from 9,415 seconds to 7,024 seconds and nearly halves GPU memory usage. As expected, the accuracy remains similar, as FlashAttention primarily focuses on optimizing attention computation by leveraging IO-awareness through GPU SRAM, as outlined in the original FlashAttention paper.

For Solana, the significance of FlashAttention is even more evident. Without FlashAttention, even a batch size of 1 cannot be processed on an 80GB A100 GPU due to memory constraints, while FlashAttention enables efficient training with a batch size of 2. These results demonstrate the critical role of FlashAttention in handling long sequences and enabling feasible training for large-scale datasets. We have updated Appendix C.3 in the paper to reflect these findings.

We thank the reviewer for the careful reading and valuable suggestions. Below are our responses:

Reviewer's Comment: The definition of anomaly transactions is ambagious.

Response: We thank the reviewer for highlighting the need for a clearer definition of anomaly transactions. Here, anomaly transactions refer to transactions that violate the design logic and usage of smart contracts and DeFi protocols. These anomaly transactions happen when attackers exploit the vulnerabilities in smart contracts and will cause serious financial losses. Given that different smart contracts may have different types of vulnerabilities, the anomaly transaction patterns will also be different. In our paper, we share a similar goal to the prior work [1] and focus on identifying transactions exhibiting abnormal behaviors in comparison to benign transactions within the selected DeFi applications. We have clarified this definition in Section 2 in the revised paper.

[1] Blockchain Large Language Models

Reviewer's Comment: The size of malicious transaction data is limited.

Response: We appreciate the reviewer’s concern about the dataset size. We would like to clarify that our model is trained exclusively on benign data to learn typical transaction patterns. During testing, the model identifies malicious transactions by detecting deviations from these learned benign patterns.

Regarding the sufficiency of 28 malicious transactions for evaluation, we acknowledge the challenges posed by this limited dataset. However, as detailed in Section 5.1, these transactions originate from well-established DeFi applications and have been carefully sampled from reputable sources, like CertiK and PeckShield. Each transaction was manually verified to confirm its malicious nature. They are real-world malicious transactions exploiting the smart contract vulnerability for the selected DeFi apps, and thus the number can be limited. Moreover, in our work, we utilize only publicly available transaction data from blockchain networks, and no private or permissioned transactions are used to make the dataset meet the privacy standard and able to open-source. Collecting a larger set is non-trivial; given the amount of manual effort needed for confirming them. Our future work will continue expanding the dataset. Note that prior work [1] identified a total of 116 malicious transactions, yet they were unable to share these transactions or their sources with us due to privacy concerns. We have added this discussion in Section 6 in the revised paper and open-sourced our datasets and models to foster further research.

Reviewer's Comment: Contract templates are wildly used in real-world smart contract development, leading to different smart contracts may offering similar or even identical APIs to the users. This could cause potential duplication in the transaction data.

Response: Thank you for raising this important point. To assess potential duplication in the dataset, we conducted a 5-gram BLEU similarity analysis (We chose 5-gram to avoid the false positives lead by indicator tokens e.g., [START], [CALL]). In the Ethereum dataset, only 0.05% of transaction pairs show a BLEU similarity over 0.7, and 0.023% exceed 0.8. These highly similar transactions may indeed result from contract templates. Given the low similarity ratio in our data, we do not consider potential duplication a significant issue. We have updated this discussion in Appendix B in our revised paper.