Zero-shot Outlier Detection via Synthetically Pretrained Transformers: Model Selection Bygone!

- time & efficiency if training dataset scales up

We’d like to draw attention that in Table 3, time for baselines include both training and inference time, as they first need to train on each dataset separately preceding inference. FoMo-0D is zero-shot, and therefore bypasses training altogether, on any dataset, and the inference for test samples can be parallelized on GPU. As such, we get at least 7.4x speed up (against DTE-NP) if training and inference time are both considered (see Table 3). If the training dataset scales up, please note that the training time of the baselines will also go up (at least linear in the dataset size). FoMo-0D inference time will grow only linearly (thanks to our router attention mechanism) as a function of training data/context size. In other words, scalability is a concern for the (training of) baselines, too, and linear-time scaling is the best one can hope for.

If the training dataset grows considerably, to an extent that hinders inference time, we can resort to strategies such as dataset distillation, which takes additional training time to learn a small number of “distilled” tokens that are representative of the whole dataset. Dataset distillation as presented in [1] is scalable, also requiring linear time-complexity in the training dataset size. Having learned the small/constant number of distilled tokens as context, inference can then be sped up considerably.

We’d like to also remark that time-critical applications often demand OD over data streams - in which it is often the case that only the most recent data is relevant (due to possible non-stationarity) and hence arguably a context size of 5K would effectively accommodate the window of relevant data in the recent history in such scenarios. While our work focused on designing the first foundation model for static OD, FoMo-0D is readily applicable to operate on streaming data thanks to its zero-shot nature that does not need re-training.

- post-training for performance improvement

Yes, even though we focused on zero-shot detection and in-context generalization, we can indeed think of a model that does more work at inference time.

A promising direction is to “attune” the model better to a specific input dataset via fine-tuning, but note that the only labels we have at inference are the inliers in $D_{train}$. Having completed pre-training, and given a new dataset D = {(inlier-only) $D_{train}$, (unlabeled) $D_{test}$}, FoMo-0D parameters can be fine-tuned to accurately predict label 0 for all the inliers in $D_{train}$, which in effect could serve as better attuning the model to the given inlier distribution prior to inference. To take fine-tuning one step further, self-training could be employed, where we can first treat predictions from FoMo-0D on new samples as pseudo-labels, and later use them toward fine-tuning the model parameters.

Of course, these approaches have different trade-offs not only w.r.t. inference time but also access to model parameters. Our work has focused on zero-shot generalization of synthetically-pretrained Transformer models for OD. It would be valuable future work to design FoMo-0D variants with varying trade-offs that could serve OD applications with different needs.

[1] In-Context Data Distillation with TabPFN. Junwei Ma, Valentin Thomas, Guangwei Yu, Anthony L. Caterini. CoRR abs/2402.06971 (2024)

- choice of baselines

Our OD baselines constitute quite an extensive testbed; with 26 different well-established methods – a much larger number than typically seen in current literature. These baselines are diverse; including both well-known shallow methods as well as deep models, and among the latter, include those based on recent techniques such as diffusion-based models from as latest as 2024. In fact, we improve over the SOTA diffusion-based approach DTE (Livernoche et al., 2024) which compared to the same set of 25 baselines.

Algorithms that “do not require retraining on the new test set” would be zero-shot OD algorithms for which the literature is slim. OOD methods do not directly apply to OD.

Please note a key difference between OD vs OOD: OD is typically given a single, fully unlabeled dataset with the goal of detecting outliers that do not “fit in” with the inliers, while OOD is provided with multiple, labeled (or known) K classes and the goal is to detect novel classes K+1, K+2, etc. In fact, Figure 1 in reviewer-provided citation [1] illustrates the differences: OOD requires ID (in-distribution) classification, while OD does not. Also, OD concerns both covariate shift and semantic shift whereas OOD mainly focuses only on semantic shift (i.e. new classes). As such, OOD baselines do not directly apply here. And of OD baselines mentioned in Section 4 of [1], many well-established ones are already included among our baselines, such as LOF, OCSVM, GANomaly, DeepSVDD, etc.

- data synthesis

Our main contribution is not data synthesis, but a foundation model for OD. In fact we use draws from GMMs for inlier synthesis and borrow ideas from existing literature (see ADBench; cited as (Han et al., 2022)) to synthesize outliers to obtain pretraining data. As such, synthetic data is only a component of our work, on which we train a novel Transformer/foundation model with in-context outlier detection ability. As far as we can tell, VOS [2] is not a pretraining-focused foundation model with zero-shot outlier detection capability.

- actual detection metrics

All of those have already been given in the Appendix - please see Table 12 (pages 34-35), Table 13 (pages 36-37), and Table 14 (pages 38-39) respectively presenting AUROC, AUPR and F1 absolute performance values on all 57 datasets for all 30 methods.

[1] Yang et al., Generalized Out-of-Distribution Detection: A Survey

[2] Du et al., VOS: Learning What You Don't Know by Virtual Outlier Synthesis

- in-depth insights on GMM prior

This outcome, that a basic data prior derived from GMMs can drive a pre-trained Transformer generalize to real-world datasets, is indeed a curious one. The emergent in-context learning (ICL) ability of pretrained Transformers on many non-trivial downstream tasks is intriguing, and has similarly surprised the NLP and ML communities at large (triggering a series of theoretical work aiming to understand how/why ICL works).

Nevertheless, there could be a simple explanation to GMMs’ prowess in our setting: which is that GMMs are good models for real-world datasets in our testbed. To test this hypothesis, we performed goodness-of-fit tests: We fit GMMs to each real-world dataset D_real with up to 5 components (as with our pre-training datasets), then sampled $D_{syn}$ from the best-parameter-fit GMM and performed a two-sample test on $D_{real}$ and $D_{syn}$ (here we used e-test [1]), with the null hypothesis stating that they come from the same distribution. A smaller p-value of such a test provides evidence toward rejecting the null (i.e. GMM is not a good fit to $D_{real}$).

We provide the results below, depicting the p-value (of the goodness-of-GMM-fit test) vs. FoMo-0D’s performance rank among 30 baselines (lower is better): We observe (Table 1) that performance is good on datasets with relatively large p-value where we cannot reject the null (i.e. GMM is a relatively good fit). This is where arguably FoMo-0D recalls its “past experience” and generalizes to datasets similar to those seen during pretraining. We also see (Table 2) datasets with relatively poor performance where we can reject the null (i.e. GMM is not a good fit). These can be attributed to falling short in generalization to out-of-distribution datasets.

Curiously, on the other hand, we also still observe (Table 3) many datasets where p-value is small (GMM not a good fit) yet the performance is competitive — those are the datasets for which FoMo-0D seems to have achieved out-of-distribution generalization; a property of ICL. It remains an open (theoretical) question to understand what (algorithm, if any) FoMo-0D might have learned that generalizes to those out-of-distribution datasets. It is also an open (empirical) quest to explore whether a more complex data prior, beyond GMMs, could further push the performance up and by how much. All in all, our work presents the first foundation model for zero-shot OD in the literature and opens new theoretical and empirical directions for the community to research further.

[1] https://www.rdocumentation.org/packages/energy/versions/1.7-11/topics/eqdist.etest

- scalability of the method, inference latency & memory overhead

FoMo-0D scales linearly with respect to its context size $n$ in both pretraining and inference – thanks to the router mechanism we employed for self-attention, that takes $O(Rn)$ instead of $O(n^2)$ where $R$ is the number of routers. Thus, complexity is effectively $O(n)$ (i.e. linear) for a small/constant number of routers (we used $R$=$500$ in our experiments).

As our paper reports (please see Table 3), inference latency is $7.7 ms$ (milli-second) per test sample on average using a context size of $n$=$5K$. As for memory overhead, FoMo-0D requires less than $1000 MiB$ GPU memory; please see line 1089, Figure 8 (middle) in Section E.2 Effect of Routers on Cost in the Appendix.

To speed up inference for massive datasets with very large $n$, dataset distillation can be employed as presented in [1]. Here, prior to inference, we’d freeze the pretrained FoMo-0D and distill a given dataset by learning a set of “distilled” input tokens, which produce similar predictions for all training samples when fed to the frozen FoMo-0D, i.e. loss is based on ALL training points (Please see Figure 2(a) in [1]). In our case, training points in D_train are all inliers with label 0, thus dataset distillation in effect would estimate a small number of representative inlier distribution tokens to be used as (now smaller) context. Our work’s main focus has been to design a zero-shot OD foundation model, and we agree that scaling improvements would constitute valuable future work.

- GMM for synthetic data generation

GMMs are relatively simple, yet they can capture increasingly complex distributions with an increasing number of components. We found them to be surprisingly effective as synthetic data prior. Our work left as future work exploring other, more complex data priors, for both inliers and outliers, since GMMs sufficed to achieve SOTA results against numerous baselines in our benchmark testbed.

For scalability to complex input datasets during inference, we tested FoMo-0D on 57 real-world datasets, including complex, high-dimensional ones, such as 5 vision/image datasets (d=512) and 5 NLP/text datasets (d=768). The results (Section G Full Results in Appendix) show that FoMo-0D performs competitively on these datasets, despite being pre-trained on relatively smaller dimensional (d=100) prior data.

- finetuning on training data

This is a good direction to explore to speed up inference in terms of absolute time. While FoMo-0D scales linearly w.r.t. context size (e.g. if context/dataset size doubles, inference time also doubles) (as discussed above), one may still aim to keep the context as small as possible for speedy inference in terms of absolute wall-clock time.

One direction in these lines is dataset distillation (see [1] above): Having completed pre-training, and given a new dataset {(inlier-only) $D_{train}$, (unlabeled) $D_{test}$}, one can learn a small number of “distilled” dataset tokens based on $D_{train}$, toward utilizing a more compact context for test samples in $D_{test}$. Note that this would still utilize in-context learning, but now with a smaller “distilled” context with reduced inference latency.

In the traditional sense of the term fine-tuning, on the other hand, one may consider parameter fine-tuning on labels, but the only labels we have at inference are the inliers in $D_{train}$. That is, FoMo-0D parameters can be fine-tuned to accurately predict label 0 for all the inliers in $D_{train}$, which in effect could serve as better attuning the model to the given inlier distribution prior to inference. Thus, fine-tuning may help improve performance, as it implicitly aligns (“distills” or “bakes in”) the given dataset to the model parameters. However, note that it does not reduce context size and hence would not help with inference latency.

More inference-time work could help offload some of the learning on a new dataset from fully-in-context to fine-tuning, and in doing so, may reduce the model’s reliance on a large context size input. Of course, these approaches have different trade-offs not only w.r.t. inference time but also access to model parameters. Our work has focused on zero-shot generalization of synthetically-pretrained Transformer models. It would be valuable future work to design FoMo-0D variants with varying trade-offs that could serve applications with different needs. Specifically, more work/compute/learning can indeed be dedicated to inference time with the aforementioned approaches (or other future work) to speed up FoMo-0D further.

[1] In-Context Data Distillation with TabPFN. Junwei Ma, Valentin Thomas, Guangwei Yu, Anthony L. Caterini. CoRR abs/2402.06971 (2024)

Thank you for your support of our work. Please let us know if you have any further inquiries about our paper and/or rebuttal.