LOB-Bench: Benchmarking Generative AI for Finance - an Application to Limit Order Book Data

We sincerely thank reviewer Nn9N for their detailed remarks and for recognizing the potential of this work in establishing a foundation for developing more robust and interpretable generative models in finance. We address the queries and concerns below, and would welcome any follow-up questions or suggestions. If these have been suitably addressed, we would appreciate an increase in the review score to help build a stronger case for acceptance.

Generalizability to Different Stocks

Our proposed framework is inherently stock-agnostic. Users of LOB Bench can evaluate their models on any asset within the LOBSTER universe – or even beyond, provided the data is converted into the LOBSTER format. For this study, we selected two stocks that differ along key metrics and are representative of assets that models could be trained on in practice.

While we acknowledge that different asset classes, such as ETFs, may yield different results, exploring this was beyond the scope of our work, which focuses on presenting the benchmark itself. We also refer reviewer Nn9N to our response to reviewer KDf5 regarding “Limited Assets” and “Zero-Shot Transfer Evaluation” for additional context on our stock selection.

Sensitivity to Different Market Conditions

We consider the evaluation of different market conditions to be distinct from the evaluation of additional stocks. A single stock can experience varying market conditions over time, including fluctuations in volatility, spread, and trading volume. Our benchmark directly accounts for this by evaluating conditional distributions.

For instance, the conditional distribution of message inter-arrival times, given the spread, captures how a model reflects variations in inter-arrival times across different market conditions (i.e., spread regimes). This approach ensures that the benchmark effectively assesses a model’s ability to adapt to dynamic market environments.

Alternative Practical Evaluation: Downstream Tasks & Sim-to-Real Transfer

We acknowledge the concerns regarding the reliance on distributional similarity metrics, such as the L1 norm and Wasserstein-1 distance. While these metrics are valuable for comparing real and generated data, they may not fully measure the degree of practical utility of synthetic data in all real-world applications.

To address this, we propose including a brief evaluation of mid-price trend forecasting—a downstream task relevant to algorithmic trading—in the camera-ready version of the paper. Specifically, we will assess model performance on a held-out set of real data after training on three different datasets: (1) real historical data, (2) a combination of real and generated data, and (3) purely generated data. This allows assessing the sim-to-real transfer gap and thus evaluating the quality of the generated model. It is worth noting, however, that this approach has its shortcomings.

The implementation will follow the model architectures proposed in Prata et al. (2024), focusing on the best-performing BINCTABL architecture, along with DeepLOB and a simple LSTM. This evaluation will serve as a complementary assessment to the distributional metrics, providing insights into how synthetic data impacts mid-price forecasting accuracy. Usefulness for predicting mid-prices might be a very discrete property of generated data, and a potential failure in this regard might prove uninformative for future model development.

RL Using Generated Data

Training reinforcement learning (RL) agents using generated data in a limit order book (LOB) setting is an active area of ongoing and future research. This approach represents another instance of sim-to-real transfer evaluation. Generative models have the potential to overcome the limitations of training policies solely on static historical data by introducing dynamic, action-dependent data trajectories, thereby enriching the training environment.

Grammar and Spelling

Thank you for highlighting these specific typos. We have now corrected them in the manuscript.

We sincerely thank reviewer EUwk for their thorough review and detailed feedback. We appreciate their recognition of the key strengths of our paper and framework, including the robust justification for our methods and scoring functions, the generality of the methodology, market impact evaluations, fully distributional scoring, and the extensive visualizations provided. We address the queries below, and would welcome any follow-up questions or suggestions. If these have been suitably addressed, we would appreciate an increase in the review score to help build a strong case for acceptance.

Discriminator Training

We do not make any specific claims or recommendations regarding the architecture of the discriminator. Our chosen model, combining Conv1D with attention layers, performs well and consistently distinguishes generated data from real data across all generative models, with a clear gradient in discriminator certainty across models indicating a varying degree of model veracity.

Rather than focusing on a particular discriminator design, we emphasize the value of using a discriminator as a learned adversarial scoring function. This approach remains valid as long as the model is sufficiently strong to learn the discrimination task effectively. Consequently, ablations on specific model characteristics are less relevant, as various state-of-the-art models could fulfill this role.

The key challenge we aim to address is improving generative model performance. In this regard, our chosen discriminator effectively exposes the gap between generated and real data while providing a meaningful learning signal for future progress.

Sensitivity to Choice of Aggregator Function

For a detailed analysis of how benchmark results vary with the choice of aggregator function, we refer to Figure 10 in Appendix D, which presents a bar plot comparing distributional errors across models. To ensure that outliers in scores do not disproportionately impact model rankings, we also report the median and interquartile range (IQR) of scores in the model summary plots (Figure 4).

Sensitivity of Divergence Metrics to the Binning Strategy

Thank you for suggesting this ablation. We are now evaluating the divergence scores with bins that are half and double their current size. We will report the sensitivity of the results based on the range of resulting scores in the camera-ready version of the paper.

Since we use a dynamic regular bin size determined by the Freedman-Diaconis (FD) rule, which is designed to adapt well to the distribution of the data, we do not expect significant sensitivity to changes in bin size. Furthermore, the following theoretical convergence argument supports this choice: the FD bin size minimizes the integrated mean squared error between the histogram and the theoretical data distribution. Bin size decreases at a rate of n^{1/3}, and the expected number of observations per bin increases without bound. This suggests that, for large sample sizes, our choice of bin size will not introduce substantial inaccuracies in the divergence scores.

We sincerely thank reviewer KDf5 for their detailed review and thoughtful comments, which not only provide valuable feedback but also highlight key strengths of our paper. In particular, we appreciate the recognition of the important gap our work addresses by introducing a well-founded, fully distributional evaluation of generative LOB data, including assessments of model derailment and price impact functions. We address the queries and concerns below, and would welcome any follow-up questions or suggestions. If these have been suitably addressed, we would appreciate an increase in the review score to help build a stronger case for acceptance.

Limited Assets

The reviewer raises a concern regarding the limited number of assets evaluated in our study. While we acknowledge that results may vary across different stocks, we selected GOOG and INTC as representative examples because they span a broad spectrum of relevant statistics, such as volatility, relative tick size (tick size relative to stock price), and traded volume.

Expanding the evaluation to additional stocks would require substantial computational resources. Training each considered model class on a new stock dataset takes multiple days on 8 L40S GPUs per stock. Moreover, as is standard practice in the domain, each model is trained on a single stock, since ample training data is usually available, and models are expected to specialize in the characteristics of individual stocks.

Small-cap and less liquid stocks, while an interesting avenue for future research, present additional challenges beyond the scope of this paper. These stocks provide less training data due to lower trading activity, which complicates deep learning approaches. A promising direction for future work is multi-asset modeling, which could leverage data across multiple stocks to enhance training and capture stock-independent dynamics.

It is important to emphasize that the benchmark itself is designed to allow researchers to evaluate models on any stock of their choice, and its usefulness is not limited to the two stocks presented in our paper. Our package includes efficient code that enables fitting baseline model parameters to new stock data in seconds to minutes. Furthermore, we plan to expand our research by developing new models that will naturally be evaluated on a broader set of stocks.

Zero-Shot Transfer Evaluation

An alternative evaluation could assess zero-shot model transfer—training on one stock (e.g., GOOG) and generating data for another (e.g., WMT). While this reduces computational costs, it introduces challenges.

First, seeding mechanisms vary by model class: autoregressive models allow conditioning on history, but the Coletta CGAN model only seeds at the start of the day, and the baseline model does not support seeding. Second, it is unclear whether performance differences in transferred data stem from stock characteristics or model limitations. A more robust approach would involve training on multiple stocks, which is beyond the paper’s scope.

That said, our benchmark supports such research directions, as it remains agnostic to the model training regime.

Discriminator Scores

We have added the following to the paper:

“High discriminability may result from model errors, as indicated by imperfect model scores [see …]. A distributional mismatch in a single scoring function can be sufficient to make fake data identifiable. To mitigate this issue, future research could evaluate adversarial performance by training a discriminator on perturbed data and reporting scores conditioned on the noise level, particularly as models improve on this benchmark.”

Low discriminator scores suggest that even state-of-the-art models still struggle to generate perfectly indistinguishable LOB data. Future improvements may follow two paths: (1) training larger supervised models on richer datasets, akin to recent LLM advancements, or (2) explicitly minimizing discriminator scores, as in adversarial frameworks like GANs (Goodfellow et al., 2014) or GAIL (Ho et al., 2016).

## Beyond LOB Data To highlight the transferability of our methodology, we added the following to the paper:

“Our methodology formalizes and naturally extends common evaluation practices for synthetic one-dimensional time series, such as financial returns, which typically emphasize distributional similarity. Our framework enables a quantitative assessment of distributional properties in structured high-dimensional time series. By adapting the scoring functions, our approach could also be applied to financial transactions, payment data, streamed price quotes in forex markets, multi-asset limit order books, or decentralized crypto market protocols.”

Documentation

We provide a Jupyter notebook demonstrating LOB Bench usage and will further improve the package documentation before the paper’s potential publication.