The paper introduces a new multimodal dataset, MBD, from over 2 million corporate clients, including bank transactions, geo-locations, technical support dialogues, and monthly aggregated purchases of four banking products. The authors also present a benchmark for evaluating models on this dataset and two other financial datasets, focusing on tasks like purchase prediction and multimodal matching.

This paper introduces a large-scale publicly available multimodal banking dataset (Multimodal Banking Dataset, MBD). The MBD contains data from over 2 million banking customers, covering four different modalities: 9.5 million banking transaction records, 1 billion geographic location data points, 5 million customer interactions with technical support embedded dialogues, and banking product purchase activity over a 12-month period. This dataset provides researchers with a rich resource for analyzing customer behavior dynamics and contributes to the development of large-scale multimodal event sequence algorithms in the future.

1. The paper introduces a new dataset, Multimodal Banking Dataset which integrates multiple modalities for over 2 million corporate clients.
2. The authors highlight the potential applications of this dataset for campaign planning and client behavior analysis, using multimodal benchmarks to demonstrate its value along with baseline model implementation.

This is not a research contribution per se but it is a dataset paper.

The paper presents the first large-scale multimodal banking dataset for the user community. It offers a new dataset that has millions of users and millions of transactions that have been suitably anonymized. Moreover, the authors have provided baselines for a few standard tasks. The dataset will be released in order to spur research in the use of machine learning for banking applications.

3. Anonymization.

Any anonymization (including ours) may remove some specific patterns in data and, hence, impact performance on some downstream tasks, but our experiment proves that our anonymization techniques still save all significant information for introduced downstream tasks. In Subsection 3.2 of our paper, we provided all permissible details of the anonymization process applied to our data. Additionally, Subsection 5.2.1 discusses the impact of anonymization by comparing model performance on the public dataset to that on the private (non-anonymized) dataset version. Figure 2 demonstrates a strong correlation between the results, indicating that the anonymization process has minimal impact on model performance for the downstream task of campaigning. This consistency underscores the reliability of our dataset for benchmarking and advancing research in multimodal event sequence analysis. Additional experimental comparison of results on our MDB and private datasets are available in Appendix B (Tables 6-9). The ranking of methods remains consistent despite differences in absolute metric values. As shown in Table 7, TrxTabGPT (ROC-AUC 0.796) and TrxAggregation (ROC-AUC 0.780) achieve the best performance on the private dataset. Similarly, TrxTabGPT leads on the public dataset with a ROC-AUC of 0.802 (Table 9). Incorporating geolocation and dialogue modalities further improves results, with DialogLast+TrxTabGPT+GeoTabGPT attaining the highest ROC-AUC on both datasets—0.802 on the private dataset (Table 7) and 0.808 on the public dataset (Table 9). Overall, multimodal approaches utilizing TabGPT or Aggregation demonstrate superior performance, with Late Fusion consistently outperforming Blending across private and public datasets (compare results in Table 7 and Table 9 to those in Table 6 and Table 8).

4. New downstream tasks.

In addition to the downstream task outlined in Section 7: Conclusion and Future Work, we plan to further develop the multimodal dataset and release an updated version in 2025. The planned improvements include:

1. Extend the time range of the dataset to include data from 2023 and 2024, enabling longitudinal studies and trend analysis.
2. Incorporate new data sources, which will predominantly be utilized as additional input modalities to enhance the dataset's multimodal capabilities. These will include:

• Expanded communication data, such as new communication channels and enriched dialogue features (e.g., discussed topics), enabling research into the impact of communication on sales and financial activity.
• Clickstream data from an internal site, covering visited pages, operations performed, and viewed advertisements, providing insights into customer online behavior.

3. Expanding Prediction Targets, which can serve as both input modalities and prediction targets. These will include:

• Additional financial products (up to approximately 20), broadening the scope of target variables for prediction.
• Customer Behavior Insights: customer churn, fraud detection, credit risk assessment
• Segmentation of customer life cycles.

These updates will enhance the dataset's versatility for various tasks, enabling more comprehensive evaluations of representations. With a multi-task framework, we aim to facilitate innovative training methods, such as zero-shot learning, and support research in financial applications and customer behavior.

5. Lack of novelty.

We agree that our paper mostly concentrates on a new dataset and corresponding benchmark rather than developing innovative models. Our submission form chose the “datasets and benchmarks” as a Primary Area. As mentioned in Section 1. Introduction, we believe that papers in this area should mainly open new research questions and provide opportunities for future studies to develop new models rather than provide a strong innovative model and close the possibilities for significant improvements. As an example, we used an article about the HINT dataset (https://openreview.net/forum?id=kIPyTuEZuAK) that was accepted as a notable-top-25% paper at ICLR 2023.

We should emphasize that our dataset forces development of new techniques rather than simply applying existing models. Indeed, as detailed in Section 3.1 (Modalities), the event sequences in our dataset differ significantly from traditional time series due to their irregular temporal patterns and the inclusion of categorical features. For example, the methods suggested in the review ([1] Learning Deep Time-index Models for Time Series Forecasting; [2] Crossformer: Transformer Utilizing Cross-Dimension Dependency for Multivariate Time Series Forecasting) are primarily designed for predicting future values of a time series, which is fundamentally different from our downstream tasks, and heavily rely on temporal continuity. So they would require substantial adaptation to effectively handle the multiple data sources and asynchronous nature of our data.

Thus, we believe this contribution will drive meaningful advancements and help bridge a critical gap in research of multimodal sequential data analysis for real industrial use-cases.

We once again thank all the reviewers for assessing our work and for their valuable comments and advice, which helped us improve its presentation. We are happy to answer any additional questions you may have.

Thank you very much for your comments, which allowed us to address the shortcomings and refine the presentation of the paper. Below we provide additional experimental results to be included in the paper. Some experiments are still in progress and will be finalized soon. We believe these updates significantly strengthen our work. The key updates are summarized below:

1. LLM-based methods for our dataset.

Initially, we experimented with several methods for integrating transactional data into LLMs, but their performance lagged significantly behind specialized baselines, so we did not plan to include these results in the paper. Using a relevant article from NeurIPS’23 (https://openreview.net/forum?id=md68e8iZK1), we conducted preliminary experiments with one of the leading LLMs, which is known to be one of the best models for the language of our dataset. We focused on transactional data from our MBD dataset. Transactions were transformed into text format, processed by the LLM to generate embeddings, and used as input for a gradient-boosting model. To speed up the evaluation during the rebuttal period, we considered predictions for the last month only. We examined two kinds of implementation: 1) the pre-trained LLM; and 2) a fine-tuned version of this LLM (setting similar to BloombergGPT). Zero-shot text-pretrained LLM showed considerably lower quality compared to CoLES, while domain fine-tuning improved performance of LLM, though it still did not surpass CoLES. An alternative method, ESQA (https://arxiv.org/abs/2407.12833), which employs a specialized encoder to structure transactional data, also failed to outperform baseline.

Moreover, we discovered that LLM methods can extract new valuable features from transactional data, as confirmed by the improved downstream task performance (up to 2% comparing to CoLES) when LLM embeddings were concatenated with CoLES embeddings (“CoLES + LLM FT” in this Table). This is a promising result, indicating the potential of LLMs for transactional data analysis. We will include these new results in the paper after recalculating them for the whole validation set described in the article, what requires additional time. Our preliminary estimates show that we need 2 weeks to reproduce the complete validation pipeline using LLMs from this Table.

2. Advanced multimodal fusion techniques.

To demonstrate the possibility of using more advanced multimodal fusion techniques, we conducted additional experiments of early fusion methods using the cross-attention mechanism, one of the foundations of modern multimodal models mentioned by the reviewers ([1] https://arxiv.org/pdf/2206.06488) and Crossformer ([2] https://openreview.net/pdf?id=vSVLM2j9eie ).

In the following table, we compared the performance of CrossTransformer (early fusion, supervised, transformer) with the results presented in our paper for the Supervised (RNN) (late fusion, supervised, RNN).

The analysis revealed that CrossTransformer performs similarly to or slightly worse than the Supervised (RNN) method. To examine the impact of Transformer architectures on the performance of multimodal data processing, we are conducting additional experiments with the Supervised (Transformer) model (late fusion, supervised, transformer), built on the same principles as Supervised (RNN). We will include the results of a comprehensive comparison of different groups of methods (late fusion and early fusion) into the paper to provide a complete and thorough comparison of multimodal methods.

We sincerely thank all reviewers for their valuable comments and suggestions. Your insightful feedback has been instrumental in refining our work. We have revised the paper thoroughly to address your comments and improve its overall quality and clarity.

Summary and Recommendation for Rejection

Strengths:

1. Dataset Scale and Scope:

   • The authors present the first large-scale, multimodal banking dataset (MBD) covering over 2 million corporate clients.
   • Data sources include bank transactions, geolocation data, customer support dialogues, and aggregated purchases.

2. Benchmarking Contribution:

   • Includes baseline evaluations comparing unimodal and multimodal methods, showcasing the advantages of multimodal approaches for financial tasks.

3. Public Availability and Reproducibility:

   • The dataset and experimental code are promised to be made publicly available, promoting transparency and fostering further research.

4. Privacy Measures:

   • Anonymization techniques are applied to protect user data while retaining significant information for downstream tasks.

Weaknesses:

1. Lack of Novelty:

   • The paper primarily focuses on dataset creation and lacks significant methodological innovation.
   • For ICLR, novel methods tailored to the dataset are expected but are absent here.

2. Sparse Details on Anonymization:

   • While anonymization is discussed, there is insufficient quantitative evaluation of its impact on preserving temporal dependencies.
   • This raises concerns about data reliability, especially for fine-grained tasks like fraud detection.

3. Limited Methodological Exploration:

   • Advanced multimodal fusion techniques, such as cross-attention mechanisms, are not thoroughly explored.
   • Comparisons with state-of-the-art methods, including large language models (LLMs), are limited.

4. Repetitive Modality Representations:

   • The modalities are primarily tabular or text-based, limiting the diversity seen in standard multimodal datasets (e.g., including audio or video).

5. Presentation Issues:

   • The manuscript contains grammatical errors, unclear expressions, and inconsistent tense usage, detracting from readability.

Authors' Mitigation:

1. Advanced Fusion Techniques:

   • The authors added preliminary results using early fusion techniques (e.g., CrossTransformer), but these showed no significant performance improvement.

2. Expanded Benchmarking:

   • Efforts to address label imbalance (e.g., stratified batching, oversampling) were included, but metric improvements were marginal.

3. Anonymization Impact:

   • Claims of minimal performance differences between anonymized and original datasets are made, but quantitative evaluations for complex tasks are missing.

4. Future Dataset Enhancements:

   • The authors promise to expand the dataset with new modalities and tasks in future versions but do not address current limitations.

Remaining Weaknesses:

1. Insufficient Novelty:

   • The authors fail to provide novel frameworks or techniques tailored to the dataset.

2. Unsubstantiated Claims:

   • Assertions regarding anonymization and dataset reliability are not adequately supported by experiments or results.

3. Limited Practicality:

   • Metrics like AUC are used as proxies for campaign success, but their real-world implications for financial decision-making are not validated.

4. Incomplete Benchmarks:

   • Comparisons with modern multimodal learning approaches (e.g., MMBench) and integration with LLMs (e.g., GPT-4 or BloombergGPT) are insufficient.

Decision Justification:

While the dataset's scale and intent are commendable, the paper falls short in several critical areas:

1. Scope Misalignment:

   • The focus on dataset release, with minimal methodological contributions, does not align with ICLR's emphasis on innovation.

2. Reliability Concerns:

   • The anonymized dataset's adequacy for sensitive tasks like fraud detection remains questionable.

3. Limited Community Impact:

   • Without novel methods or robust comparative analyses, the paper's contribution is constrained to the dataset.

4. Quality Issues:

   • Presentation flaws and lack of clarity weaken the submission.

Recommendation:

Reject. While the dataset provides a strong foundation for future research, significant revisions and expansions are necessary to meet the standards of a premier conference like ICLR.

Reject