LOB-Bench: Benchmarking Generative AI for Finance - with an Application to Limit Order Book Markets

Clarifications

We believe there may have been a misunderstanding regarding the central scoring methodology used by LOB-Bench. To clear this up, we added additional discussion on the methodology, a new Figure 1 to illustrate the process and explicit paper contributions in the introduction.

Replies to Concerns

just having stylized facts, already thoroughly explored by Vitrienko et al. [5], isn't sufficient. While they support this claim by introducing the L1 as a qualitative metric, it lacks substance and overpresenting these stylized facts overshadows the "novelty" of the benchmark.

The point of a benchmark is to provide a theoretical framework of how to measure generated data quality, as well as, a useful practical tool for further model development. The statistics explored in Vyetrenko et al. are amongst those relevant for financial practitioners and users of such models, and are therefore necessary elements of a good benchmark. In our paper, we also show that all such statistics neatly fit into the theoretical framework of scoring functions as dimensionality reduction operators. Because models in this domain have so far only managed to qualitatively match "stylized facts", we argue that further model development is currently limited by exactly quantifying how well the model matches the actual data distribution, rather than only saying qualitatively that a model can reproduce some expected behaviour or some emerging characteristic has the expected magnitude. And that we provide by quantifying distributional similarity, e.g. by using an L1 distance.

Furthermore, we show that our framework is general enough to also encompass adversarial scoring functions, derived from GAN training. So, while statistics of interest to practical applications are essential, they are not of sole relevance.

Besides our theoretical contribution, we would argue that a benchmark's value mainly lies in its usefulness as a practical tool and ability to provide a measuring stick to further future model development.

I argue that it'd be simpler and more than enough to train a model -- even a simple one -- on a financial downstream task -- say mid-price prediction -- on "fake" generated orders and measure its MSE. Then, at test time, one would measure the model's MSE on real orders and see how well the model trained on generated orders generalizes over the real data. This would be a sufficient indicator to assess the goodness and realism of the generated data.

This is an interesting suggestion, which we had also considered. However, there are a number of issues, which unfortunately make this approach impractical.

• A significant sim-to-real transfer gap can be expected even for very good generative models, if the "simple" model trained to perform the task is too simple. Many financial applications, such as forecasting are notoriously hard to do well, requiring a lot of development work, which makes such a benchmark complex to perform.
• Even using a strong model, trained on good generative data will have some transfer performance loss as the model cannot be perfect. So comparing like-for-like would require comparing real-to-real with sim-to-real transfer performance, requiring two models to be trained for each task, making such a benchmark even more complex.
• More damagingly, any concrete downstream task can only evaluate a very narrow range of a model's abilities. Mid-price prediction, for example, could never quantify a model's ability to accurately model behaviour deeper in the book, which could be very important for other applications. Furthermore, using MSE as an evaluation metric wouldn't even measure very simple distributional characteristics, such as variance, and would therefore be a bad choice. Such an approach would therefore never be a sufficient indicator to assess realism.
• For many current generative LOB models, sim-to-real transfer would simply fail because the model performance is not strong enough yet. Whether a model can perform a concrete task or not can also be a very discrete difference. Such failure would however be very hard to quantify, and therefore not be as useful in a benchmark, which is aimed at gradually improving model performance.

For these reasons, we argue that our approach of measuring distributional differences is the clear better choice to accelerate model development at the current stage of research.

So what is the AUC-ROC for Coletta and the baseline? You still didn't give me an answer here.

Still using LOBSTER data can't be accessible by all research institutions. In fact, all papers in financial time series and LOB market simulations rely on third-party data which makes these papers not reproducible. What about releasing a LOB dataset that comes for free with the benchmark, like the FI-2010 dataset but with the message book? That would be something that would actually accelerate the way to better generation models.

Yeah, just saying sorry about having missed the clean link in your paper doesn't smooth the breach of anonymity. I'm upping my score to 3 (reject, not good enough), but will still argue to get it rejected.

In my paper (TRADES: Generating Realistic Market Simulations with Diffusion Models), I did what the reviewer suggested. I have calculated the predictive score for generated data, using 2 of the same models that LOB-Bench uses. Just to be clear, I did this at the end of 2023 (proof commit date on my GitHub), but I decided to publish the paper only 2 months ago.

Clarifications

Compared to current quantitative metrics in the literature, such as cross-entropy, LOB-Bench is claimed to better deal with issues like distribution shifts and autoregressive gaps within data.

Our goal for this benchmark is to be agnostic to the model class generating the data. One consequential limitation is that we cannot use cross-entropy, as not all model classes generate explicit distributions over tokens, or the generated data sequence more generally. For example, agent-based simulations or CGAN models may only generate a single data realisation for a given random seed, rendering cross-entropy calculations infeasible. We have added this point to the paper.

A key contribution of LOB-Bench is the evaluation of generated distributions, conditional on the unrolling step (see p. 17, Figure 11). This allows an evaluation of distribution shifts due to autoregressive error divergence.

Replies to Concerns

This paper displays several signs of incomplete editing that require further proofreading. Specific issues include a non-functional GitHub link on line 033, and repeated sentences between lines 097-099. Before submitting, the authors should conduct a thorough proofreading pass, paying particular attention to consistency in links and removing any redundant text.

We thank reviewer trTG for pointing out editing mistakes that slipped through the editing process, which have now been fixed in the updated manuscript.

Although the paper introduces a new benchmark for evaluating generative AI models, it does not adequately compare this benchmark with existing alternatives.

A major reason for creating this benchmark was that no comparable benchmark for generative LOB data exists yet. But to provide some more general context, we have added a discussion of other benchmark methodologies for financial data to the paper.

Furthermore, the experiments fail to convincingly demonstrate the benchmark's claimed ability to effectively address distribution shifts or autoregressive gaps within the generated data. [...] Specific experiments that clearly demonstrate superiority in dealing with distribution shift or autoregressive gaps in the generated data would make the method more convincing.

:question: Could you kindly clarify what kind of distribution shift you think we are not addressing adequately?

We are addressing two types of distribution shift directly in the benchmark: conditional distribution shift and autoregressive distribution shift. By allowing the evaluation of arbitrary conditional distributions, the benchmark can evaluate how a distribution of interest shifts, conditional on another scoring function. For example, $p(\text{spread} | \text{volatility})$ allows evaluating how well the model is able to learn this type of distribution shift of the spread. Even more generically, $p(\text{spread} | \text{time})$ captures distribution shift of the spread by time of day (see p. 18, Figure 12).

The second kind of distribution shift captured by the benchmark is autoregressive distribution shift during inference. The benchmark measures how quickly generated distributions shift from the target distribution, supporting future model development to mitigate this type of failure mode.

Since we unfortunately cannot provide raw data with the benchmark due to licensing limitations, the selection of data to use with the benchmark is up to the user, since we also do not know which data their model was trained on. To evaluate a distribution shift between train and test, users could e.g. train their models on data before 2020 and evaluate during 2021, when market behaviour may have differed due to the Covid turbulence.

We have clarified our discussion of these points in the paper.

Replies to Concerns

The paper explores generative models; some more, like advanced deep learning architectures, could be explored.

The GenAI literature for structured financial data, such as LOBs, is still a nascent field. We picked two recent SotA "advanced deep learning architectures" in different classes (CGAN, autoregressive), as well as a more classical statistical model for evaluation. While not a wide variety of advanced deep learning models have been published in this domain, we also do not claim to have performed a comprehensive benchmark evaluation of all available models. Instead, our aim is to provide this benchmark as a tool to further aid the development of new models and to provide a unified solution for comparable data formats in the field. Furthermore, comparing two SotA models, we make the case that autoregressive models appear to have an edge in this data modality and suggest this as a promising direction for further research.

The evaluation highlights that errors accumulate over longer sequences. The benchmark does not fully address how to mitigate this.

We would argue that a benchmark does in principle never take the roll of providing solutions how to mitigate types of model errors. Instead, by elucidating specific failure modes, such as error accumulation, the benchmark can aid in developing new models and methods aimed at solving such issues.

In principle, methods reducing errors on a sequence level, or RL finetuning on downstream tasks could be viable options to improve models in this direction.

Certain model results with ra[r]e events like market orders have data sparsity. and generating additional data might not be practical.

We have not yet run into problems with data sparsity, particularly with market orders. Besides, the LOBSTER data standard does not differentiate between executions due to executable limit orders and market orders. In any case, execution order types are frequent enough for generative models to learn them. Generating more synthetic data is also always possible in principle. Moreover, current generative models for LOB data still have problems learning distributions accurately, even when ignoring outliers.

Replies to Questions

Q1 The framework does not account for hidden liquidity, which is critical for LOB modeling. The model can be improved in this area.

We used the same data pre-processing pipeline for the training of all models in the paper. We indeed remove all hidden order types (type 5 in LOBSTER), which denote the execution of an incoming order against a hidden liquidity block at the end of a queue. By removing this altogether, we do not claim to model hidden liquidity but create an equal footing for each model. The GenAI task is hence to model LOB behaviour with the effects of hidden liquidity clearly removed.

:question: If reviewer GuKS has concrete examples of cases where omitting hidden liquidity in this way would affect results, we would welcome such suggestions. To the best of our knowledge, there are no models for generating market-by-order data which also attempt to model the hidden liquidity.

Q2 Why did you focus on autoregressive models, (C)GANs, and agent-based models? Have you considered other deep learning architectures for LOB data generation, such as transformers or diffusion models?

A key feature of the benchmark is that it is agnostic to the model class, which generates the data. All that is needed for compatibility is that the model generates data in the LOBSTER data format. We evaluate a (C)GAN and an autoregressive state-space-model, as these are two SotA models in the field, and a more classical statistical model which is commonly used in the finance literature. We further mention agent-based models, as these are also commonly used in the literature and have added corresponding references to the paper. Developers of transformer based sequence models or diffusion models of LOB dynamics could also evaluate their novel models using LOB-Bench. Generally, LOB data has a structured format and is not usually compatible with out-of-the box sequence models for a various reasons, such as non-stationarity, arbitrary order ids, nanosecond time precision etc.