Inference, Fast and Slow: Reinterpreting VAEs for OOD Detection

As illustrated in Table 1, the performance improvement of the proposed algorithm is not substantial when compared to other competitors. In some instances, its performance is even inferior to other state-of-the-art (SOTA) methods.

We acknowledge that our method does not outperform all SOTA methods in every case. However, LPath-1M-COPOD or LPath-2M-COPOD is competitive itself against most methods in most settings, and the union of all three of the last rows, being the new method, outperforms other methods.

Our goal is to introduce a novel method that balances performance with efficiency in model size and computational resources. We believe the LPath approach offers a valuable trade-off and can inspire further exploration in OOD detection methods.

Furthermore, the datasets used in the experiments are all small in size. Consequently, the benefits of the proposed likelihood path principle and the associated algorithms have not been fully demonstrated. It remains uncertain whether the proposed concept and algorithm would be effective in more complex real-life scenarios.

We appreciate your concern about the scale of our datasets. e acknowledge that the datasets and models in this paper are relatively small by today’s standards. However, our main goal for this work is not to show a significant performance gain, but rather to show that it is possible to use the LPath principle to achieve competitive unsupervised OOD detection performance even with much smaller models.

The datasets we selected are widely used in the literature and include challenging cases like HFlip and VFlip. These are difficult scenarios where the ID and OOD data differ minimally. Demonstrating effectiveness in such cases highlights the potential of our approach.

We agree that testing the LPath method on larger, real-world datasets is important. Given the current challenges in unsupervised OOD detection, where achieving an AUC significantly above 0.5 is non-trivial, we consider our contributions a meaningful step forward. Future work will explore the applicability of our method in more complex scenarios.

Further, we are not trying to write a purely outcome oriented paper that pushes up the benchmark by a few more points through more tweaking, we are trying to write a concept and method-oriented paper that aims to present a novel concept that aims to inspire more novel and interesting ideas.

Questions:

The proposed LPath principle and algorithm are presented in a context of Gaussian VAE. Is it possible to extend them to Gaussian mixture VAE or hierarchical VAE? By doing so, likely it can improve the OOD detection performance, while at a cost of increasing the model size.

Yes, extending the LPath principle to Gaussian mixture VAEs or hierarchical VAEs is feasible and could potentially enhance OOD detection performance. However, this would involve increased model complexity and goes beyond our computational budget. We believe this is a promising direction for future research and could further validate our approach.

As shown in Table 1, for some cases, DDPM performs much better than the proposed methods. Why? Is it because the model capacity of proposed algorithm is too small compared to DDPM? How about enlarging its model size, then compare it again with DDPM?

DDPM indeed performed better than our method in 3 out of the 9 cases, but only marginally. And we agree it could be due to the model size (Ours at 3M vs. DDPM at 46M). We note that despite this disparity in parameter counts, we managed to outperform in the Cifar-10 (ID) case and comparable in the FMNIST (ID) case. No other VAE based method comes close.

While we are seeing many versions of the scaling laws in various domains, the main point of this paper is not to scale OOD detection or to achieve the best SOTA OOD performance in every case, our paper is not outcome oriented, but rather method-oriented. Though both types of papers have their utility, we are trying to write a concept and method-oriented paper. We aim to present a novel concept that may inspire more novel and interesting ideas, instead of an outcome-oriented paper that pushes up the benchmark by a few more points through more tweaking or using larger models.

The paper is not an easy read as it quickly gets rather convoluted. For example, some efforts are spent on talking about "slow and fast weights", but it is unclear how this concept is related to the actual work. The actual work seems to be about sufficient statistics, so it seems like a detour to talk about "fast and slow weights". I may have missed an important insight here, but I do not see what the fast-and-slow-analogy brings to the table (beyond confusion for this reader).

Thank you for your feedback on communication. Our intention was to establish a foundation for applying the sufficiency principle within deep generative models (DGMs). Specifically, we observe that the likelihood function with respect to the slow weights (the neural network parameters) is intractable (as noted on lines 304–307), making it challenging to extract sufficient statistics directly from them.

Our key insight is that we can apply the sufficiency principle to the fast weights—the parameters $\mu$ and $\sigma$ in Gaussian VAEs—which are tractable and directly influenced by individual data points (as illustrated in Figure 1). This distinction is crucial because it enables us to extract sufficient statistics from the fast weights, facilitating effective OOD detection through their density estimation.

Another example of convoluted writing is that the method is motivated by the proposed "likelihood path" principle in Sec 3, but this principle is only discussed in Sec 4. It feels to me like the order of presentation is unconstructive. (I understand that such comments are fundamentally subjective, but I, at least, found the presentation confusing).

The likelihood path principle relies on likelihood and sufficiency principles, which in turn rely on slow and fast weights for DGMs. We would like to seek constructive feedback on the order of presentation. We were also worried about this issue, so we tried to include a very brief summary in the introduction, hopefully to give readers a rough idea of what it is. We’d like to hear your feedback on whether the introduction section is clear to you, or any suggestions on how a better order might look like given the dependencies we described above.

The phrased likelihood path principle is imprecise: the first sentence say that there is "more information", but it does not say relative to what. It does not help that the principle is repeated three times throughout the paper.

We appreciate this feedback and agree that the phrasing could be more precise. We also highly value being precise in the statements and claims we make. In Section 4.3, we formally define the LPath principle in Equations (23) and (24). Specifically, Equation (24) shows that when density estimation is imperfect, the mutual information between the sufficient statistics $T$ and the sample $x$ is greater than that of the marginal likelihood $p_{\theta}(x)$. In other words, $T$ contains more information about $x$ relative to $p_{\theta}(x)$.

We will revise the manuscript to explicitly state that the "more information" is relative to the marginal likelihood $p_{\theta}(x)$, addressing the ambiguity you pointed out. Our experimental results in Table 1 support the LPath principle by demonstrating that leveraging the sufficient statistics $T$ leads to better OOD detection performance than using the marginal likelihood $p_{\theta}(x)$ alone. This is significant because practical estimations of $p_{\theta}(x)$ are inherently imperfect, making the LPath principle broadly applicable.

Following the point above, then equations 23 and 24 appear incorrect. They would be correct for the ELBO or the one-sample-estimate of the marginal likelihood, but they are incorrect for the marginal likelihood (which is what the equations state).

We appreciate your observation and apologize for any confusion regarding Equations (23) and (24). On lines 364-365, we clearly stated lines 366-374 are about the marginal likelihood estimate, not but the theoretical marginal likelihood computed from integration. As you noted, the argument is correct when it is ELBO or one-sample-estimate. Our main thesis is that the Likelihood Path (LPath) contains more information and can outperform practical marginal likelihood estimates used in OOD detection.

While analyzing the theoretical marginal likelihood is interesting, it is not the primary focus of our work. Nevertheless, we can extend our argument on lines 366–374 to cases where we marginalize over $z$ through sample summation or integration. Since Equations (23) and (24) hold uniformly for every $z$, we can obtain an averaged version of these equations. We will include this extended discussion in the Appendix to provide additional clarity.