PQMass: Probabilistic Assessment of the Quality of Generative Models using Probability Mass Estimation

Thank you for your review and very detailed comments. We have attempted to answer each of them below.

The idea is pleasingly simple for a relevant contemporary challenge in stats/ML, and explained pretty clearly. The experimental results are strong and quite convincing, especially in higher-dimensional problems without an obvious feature representation. The figure quality is high throughout, which is a good thing in general. The experiments themselves are pretty thorough, as are the descriptions of the experiments, especially the astrophysics experiments that might be hard for a non-expert audience. The example detecting the small additive noise is nice in Appendix F.2. Demonstrating the method can find an effect invisible to the naked eye is good.

We appreciate the reviewers kind words, and are very grateful for their feedback

The very high figure quality has the probably unintended consequence that the resulting pdf file size is very large (>17MB), which makes it kind of a pain to handle by conventional means. If there were some way to reduce this without overly compromising readability, then that would be good. I suspect that you have one or two figures in the appendix that are unnecessarily high resolution when printed out, but I haven’t figured out which ones (possibly figures 17/18)

We apologize for our oversight. It appears that figures 17 and 18 were indeed partly to blame, as well as 12. We have reduced the size in the updated version.

A few small language issues: “Miscrepancy” line 356. I assume you meant “discrepancy”? English in general pretty good. “The instrument has multiple sources of non-Gaussian additive noise, including… Gaussian noise” is kind of odd.

These are, indeed, grammatical errors. We thank the reviewer for pointing them out, and have corrected them.

Figure design Figure 6 is hard to understand immediately. The claim about “strongly correlate” (line 446) does not jump out intuitively from the data plotted.

That is reasonable, this is indeed a busy figure. We have softened the statement about the correlation, since we agree that while the correlation is visible, the word "strong" might be exaggerated.

You call this a “Likelihood-free method”... but, you do use a multinomial distribution

Thank you for the opportunity to clarify this point:

In the context of generative models, "likelihood-free" has been used to mean simply "not using the density of or estimated by the generative model" (but instead using statistics computed from samples, as PQMass does). Likelihood-free tests, such as PQMass, FID, etc., can be used with generative models where exact density evaluation is intractable, such as GANs and diffusion models, and are thus favored over data log-likelihood in many applications.

This is distinct from another meaning of "likelihood-free" in machine learning, which is related to simulation-based inference (i.e., posterior estimation where joint probabilities are not available but ancestral sampling is possible).

In neither case does the term imply that probability computations are not involved in the test.

“PQMass works with the intervals of pdfs, it does not make assumptions about the true underlying density field”. I mean, the cdf is defined by the pdf, and assumptions about one will imply assumptions about the other, so I don’t think this argument is particularly strong

We suspect this may be a propagation of the small misunderstanding from the previous point. We simply meant that PQMass does not make assumptions about the approximability of the two test distributions by some functional form, as some other methods do (e.g., FLD relies on a Gaussian mixture approximation and FID on a Gaussian approximation).

“These results show that, at least for “well-behaved” densities, no information is lost by the PQMass statistics” Did you not say at the start that you didn’t make assumptions about the underlying density?

The lack of assumptions refers to approximability (e.g., by Gaussians or Gaussian mixtures). The assumptions of smooth density and full-support reference point distribution are very weak (and can probably be relaxed further). For example, they hold for the convolution of any distribution with a small Gaussian. We will revise the statement to make clear what is meant.

“No information is lost” in what sense? Clearly information is lost when you quantise. Is this claim meant in some asymptotic limit?

This means that if two distributions are distinct, then the test has a positive probability of distinguishing them (and the probability approaches 1 in the asymptotic limit).

This contrasts with tests that approximate the distributions by a certain fixed class. For example, if two distributions have the same second-moment statistics in feature space, FID will not distinguish them, even in the asymptotic limit of infinite samples, while PQMass will.

More explanation of the 2-d Gaussian mixture model and the 10-d funnel distribution would be good rather than just a reference, if you can find the space. Appendix is always an option.

These are both synthetic distributions used for benchmarking sampling methods. They are chosen because of the multimodality of the first, and the difficult to model shape of the second. We have clarified this in Appendix C, which also shows plots of both distributions.

Methodological questions: Is there any particular reason that you pursued Pearson’s chi squared test ahead of Fisher’s exact test, which should be less vulnerable to small contingency cell counts, which could easily happen here with the randomised voronoi tessellation?

The main reason is that Fisher's exact test, at least traditionally, is used on $2\times2$ contingency tables, whereas we have $N\times2$ tables. As far as we are aware, extending Fisher's test to an $N\times2$ case would require computing a combinatorially large number of $2\times2$ tests.

Choice of centres Is it really hard to do better than random? Some sort of relatively simple diversity measure would get you quite far here, i reckon “Sample the reference points from a uniform mixture of the empirical distributions of samples from p and q”

We have tested a variety of sampling schemes for the reference points. Our ultimate choice of sampling reference points from the mixture of p and q is reliably efficient and avoids pathological cases (see the second table included in the response to Reviewer 6iXh).

What if one of the distributions is known to be more accurate representation of reality and therefore give a better coverage of realistic regions of probability space?

Our method simply estimates the probability that both sets of samples are drawn from the same distribution, and therefore has no perception of reality, or ground truth. If one of the sets of samples has better coverage than the other, and this difference is significant enough, our method can point out that these samples come from different distributions.

“Perform the test multiple times”? How many is necessary? What are the computation implications? “

More repetitions give a better sense of the uncertainty in the estimation, but there is an obvious linear scaling of computational costs with the number of repetitions. In our experiments, we empirically found 20 to be a good enough number to estimate this uncertainty.

“This effectively marginalizes the choice of tessellation”: this comment makes me wonder whether a Bayesian formulation of the problem would work well, since you are effectively having sample the latent allocation of the points to reference centres vs cell populations. There will be analytical conjugate Bayes solutions to the multinomial distribution comparison too, so there should be no more pesky posterior sampling in addition to what is being done already

Indeed, if we assume a ${\rm Dirichlet}(\alpha)$ prior over the probability masses of $n_R$ regions in a tessellation, and observe count vector $k_P=(k_{P,i})_{i=1}^{n_R}$ of points in these regions, the posterior distribution over the masses is Dirichlet-multinomial.

The posterior predictive probability of observing an independent sample of points with count vector $k_Q=(k_{Q,i})_{i=1}^{n_R}$ is then

${P}({k}_Q\mid {k}_P)$

$=$

$\frac{\Gamma(n_R\alpha+m)\Gamma(n+1)}{\Gamma(n+n_R\alpha+m)}\prod_{i=1}^{n_R}\frac{\Gamma(\alpha+k_{P,i}+k_{Q,i})}{\Gamma(\alpha+k_{P,i})\Gamma(k_{Q,i}+1)},$

where $m=\sum_ik_{P,i}$, $n=\sum_ik_{Q,i}$ are the numbers of samples.

Although this gives us posterior probabilities, it leaves us with the problem of choosing $\alpha$, as the properties of the distribution from which the reference points defining the regions are drawn will affect the distribution of masses.

We thank the reviewer for their helpful comments.

A weakness of this study is the limited comparison with other two-sample test methods. The authors simplified the problem by dividing the data space and reducing it to a multinomial distribution comparison. However, various other methods have been proposed for distribution comparison not only MMD and W2. For example, [Ref1] explores distribution comparison using classifiers and its application to GAN evaluation, while simpler approaches, like those using nearest neighbors, are also available [Ref2]. Although the simplicity of the proposed method is an advantage, comparing its performance with these existing distribution comparison methods would be essential for evaluating its practical utility.

We thank the reviewer for raising this point. We agree that comparison with the nearest neighbors based method is useful, and we have added this to one of our experiments.

We provide an experiment comparing the performance of the nearest-neighbor two-sample tests, both unweighted and weighted, against PQMass. Specifically, we consider two distributions, $\mathcal{N}(0, 1)$ and $\mathcal{N}(0.5, 1)$, where samples from these distributions should be identified as out-of-distribution relative to each other. Using 50 samples from each distribution in two dimensions, we evaluate the unweighted test statistic $T_{k,n}$. The expected value is $T_{k,n} = 49.4900$, and the observed value is $T_{k,n} = 52.4000$, with a p-value of $p_{T_{k, n}} = 0.5596$. These results indicate that the unweighted two-sample test fails to reject the null hypothesis and incorrectly suggests the distributions are in-distribution. For the weighted two-sample test, the expected test statistic is $U_{k, n, w}$ = 1.1300, the observed value is $U_{k, n, w}$ = 1.1245, and the p-value is $p_{U_{k, n, w}}$ = 0.7055. Again, the results do not indicate a significant difference between the distributions. In contrast, when running with PQMass using 10 regions (due to limited samples), the expected $\chi_{PQM}^2$ is 9; however, the actual $\chi_{PQM}^2$ is 15, confidently indicating the data does not come from the same underlying distribution.

However, we do not believe that comparing with a classifier-based method is the most sensible approach here. The reason for that is that our method is in most cases, used to check the quality of samples from a deep generative model. Therefore, we believe that using a neural network to validate the quality of samples from another neural network is a risky approach (e.g. both generator and test networks may share a failure mode). Such a network based two sample test will share many of the challenges of GANs, and likely raise many further questions such as if one selected the ideal architecture for the discriminator, if it was trained long enough, hyperparameters tuned correctly, and so on. It seems to us that methods based on statistics, such as the one we propose here, and the one introduced in [Ref2] are more appropriate for this problem setting.

What are the strengths or advantages of the proposed method compared to other distribution comparison techniques beyond MMD and W2?

We attempted to address this in section 3.2. The main advantages of the proposed method are better scalability to higher dimensional spaces, and to higher numbers of samples than W2 and Radial Basis Function MMD (Fig. 4) and higher sensitivity to missing modes than Linear Basis Function and MMD (Fig. 3).

Practical implementation

As far as I can tell, you've only considered exactly balanced samples each time: m=n. While in some applications the user has control over both m and n, this will not be the case in many scenarios. So how well does the test perform when the two samples are imbalanced, including extreme cases?

We have multiple experiments in which we have imbalanced datasets in Appendix F.4, in which we show PQMass performs well even when the samples are imbalanced. We have included the number of samples being used in each of the experiments for clarification. In the default implementation, PQMass draws reference points uniformly from the concatenation of the two samples, so the smaller sample is likely to have fewer references drawn from it.

Using reference samples z_i randomly taken from the two distributions (x_i) and (y_i) has a long history, even when comparing samples from two MCMC samplers (or the same MCMC sampler twice, when considering convergence diagnostics; e.g. Sisson & Fan (2007). Statistics and Computing, 17, 357-367). So this approach is fine. However, a disadvantage of this is that when p and q differ only, say, by a small mode with low weight in the tail, then by sampling uniformly over the x_i and y_j, one is reasonably likely to miss samples in this mode, as the sampling will naturally favour repeated samples in areas of higher density, and hence the test will miss the obvious difference between p and q.

We should note that missing a small mode as described would affect any two-sample test by construction. One can always design a mode to be "sufficiently small" that it eludes a sample-based test as samples from it would not appear among the test samples. It is also worth noting that in the Voronoi binning step of the PQMass test, all samples are binned, such that PQMass can be sensitive to a small missing mode even without a reference sample within said mode (though it certainly helps).

However, we agree with the referee that it is important to evaluate the effectiveness of PQMass at detecting a discrepant small mode, and compare it to other existing methods. To this end we have added a test using CIFAR-10 which compares the sensitivity of PQMass with other tests used for generative models to detect a small mode.

We select eight classes (airplanes, automobiles, birds, cats, deer, dogs, frogs, and horses) and split the dataset into two sets $X$ and $Y$, each containing 24,000 samples (3,000 from each class). To simulate a small mode, we introduce a parameter, $\alpha$, and add $3000\cdot\alpha$ images of a new class (trucks) only to $X$, while randomly removing the same number of images of other classes to keep $X$ and $Y$ the same size. We then run various tests to compare $X$ to $Y$.

In the table below, we see that as we include images from the new class, which introduces a small mode, PQMass picks up the difference bewteen $X$ and $Y$ even for small $\alpha$. On the other hand, FLD struggles to detect the difference; FID also detects the difference (as shown by the monotonically increasing test statistic), but, unlike PQMass, does not give a point of reference allowing to compute significances (p-values).

We appreciate the reviewer’s comments and the opportunity to clarify the contributions of the paper. While it is true that the chi-squared test is a well-known statistical method, the specific way that PQMass has been designed to use a chi-squared test to assess the performance of generative models of different modalities in high dimensions is entirely novel.

Recent publications of methods like FID and their growing use demonstrate that there exists a clear and pressing need in the community for such methodology.

PQMass addresses this very need. While PQMass is computationally simple, its simplicity should not be equated to triviality. PQMass is a novel method that provides a reliable, interpretable, broadly applicable, and inexpensive tool to test machine learning models without the use of other machine learning models.

We believe that PQMass can be a highly impactful method. Its publication will certainly help the community learn about this method and allow it to positively contribute to the field.

On consistency guarantees

a) In Proposition 1, we require $m$ and $n$ to grow at rates such that Fisher's test for multinomial distributions with $m$ and $n$ samples from the two distributions distinguishes them. This is guaranteed by $m/n$ being bounded both above and below by strictly positive, finite scalars. We will clarify this is the meaning of the limit in the text.

b) $\pi$ depends on $r$, so the probability referred to in Proposition 1 is over the choice of $n_R$ samples from $r$. Of course, for "bad" choices of reference points, we may be unlucky and not distinguish $p$ and $q$ (if the two resulting cells have the same mass under $p$ and $q$). The proposition states that with positive probability, this does not happen.

c) Proposition 2 is indeed about the asymptotic limit. Statistical tests comparing a model to ground truth data assume that the latter is sampled i.i.d. from some underlying distribution, even if we only have access to a finite number of samples. The proposition simply tells us that the sensitivity of the test approaches 1 as the number of samples grows, not the rate. The usefulness of the test itself is shown by the empirical evidence in our experiments.

d) We clarify two points:

• First, "no information is lost" means that if two distributions are distinct, then the test has a positive probability of distinguishing them (and the probability approaches 1 in the asymptotic limit). This contrasts with tests that approximate the distributions by a certain fixed class: for example, if two distributions have the same second-moment statistics in feature space, FID will not distinguish them, even in the asymptotic limit of infinite samples, while PQMass will.
• Second the $n_R\to\infty$ limit in Proposition 2 does not refer to performing the test with "$n_R=\infty$" to compare two chi-squared distributions with infinite degrees of freedom. The proposition states that the probability that the tessellation defined using $n_R$ random reference points sampled from $r$ separates $p$ and $q$ -- which is a well-defined quantity for any $n_R$ -- approaches 1 as $n_R\to\infty$.

We thank the reviewer for their very helpful comments. We have tried to address all the issues raised to the best of our abilities.

The use of L2 or L1 distance metrics may limit PQMass's effectiveness, especially when the data resides on a complex manifold. These metrics may not capture meaningful differences in such cases, potentially leading to inaccurate assessments.

This is a great suggestion. To address this, we have decided to expand the discussion of the choice of the distance metric. We have added an extra appendix (Appendix G) to discuss this. In the extra experiments presented here, we have explored the performance of PQMass using 8 different common distance metrics. All metrics perform as expected, resulting in a $\chi^2$ distribution for the null test (Figure 20 and Figure 21, left panel) but show different amounts of sensitivity for differentiating two distinct distributions, as you pointed out. We also repeat the experiment in Section 3.3 -- adding noise to ImageNet -- with the same eight metrics. In Figure 20 we show that all metrics correctly result in the $\chi^2$ distribution for the null test. We also show the metrics as noise of different variance is added and showcase that the choice of metric will have an effect on results.

Note that although a suboptimal choice of distance metric could lead to a less informative statistic (in the sense that the test could potentially not differentiate two distributions with confidence), the test will not falsely rule out the null hypothesis. Our general suggestion is to use the Euclidean metric by default, but in cases where this metric is not sufficiently informative one could choose another appropriate metric, such as one defined in feature space.

Experimental evaluation mostly focus on synthetic data or standard datasets like MNIST and CIFAR-10. Testing on more complex, real-world datasets would strengthen PQMass’s claims about the performance of the proposal.

We also felt that standard datasets were not enough to test this method (though we would like to point out here that many competing methods were only tested on these benchmarks when introduced). This is why we included tests on higher-dimensional benchmarks like ImageNet and presented experiments on real-world complex datasets from the field of astrophysics and tabular census data. These can be found in Appendices F.3 and F.4, and we are happy to point to them more prominently in the main text.

Although the paper briefly addresses other data types, it primarily evaluates PQMass on image data. Expanding experiments to more complex, structured data (e.g., text or biological data) and analyzing the impact of different distance metrics in these contexts would help to better validate PQMass performance.

Again, other data types were added to the appendix due to space limitations (see Appendices F.2 and F.3). In addition, and in response to the referee's helpful suggestions, we have added a new experiment on protein sequences to Appendix F.5. We believe that this experiment addresses all three weaknesses addressed by the referee: It does not use the L2 distance metric, it is a complex real-world dataset, and uses complex, structured data.

I would like to hear authors' opinions about the detailed weaknesses of their proposal.

We tried our best to address the weaknesses we found in the limitations section (sec. 4). However, if the reviewer feels like there are any missing limitations in that section, we would be happy to add them.