Kernel Trace Distance: Quantum Statistical Metric between Measures through RKHS Density Operators

We sincerely thank the reviewer for their thorough and thoughtful review and his very positive comments on our paper. We answer his questions below.

1. We chose the parameter $\sigma = 1$ in both experiments for practical reasons: in the particle flow setting, this value matched the scale of the image data. While we could have used the median heuristic (i.e., the "median trick," based on pairwise distances), it yielded a similar value. In the ABC experiment, $\sigma = 1$ was also natural, as the models involved Gaussian distributions with standard deviation one.

2. See answer to Q1/Q2 to Reviewer vtMw.

3. Building on the previous point, there also exist methods to approximate eigenvalues more efficiently. Since our analysis assumes that only the top eigenvalues are most relevant, such approximations could be a valid way to accelerate computation, though they would reduce the precision of the distance estimate and weaken theoretical guarantees.

4. We grouped the two experiments together because the first provides intuition that helps in understanding the second, more complex case. However we agree with the reviewer suggestions and have reorganized this section.

5. The proof for MMD follows a similar structure: it begins and ends in the same way, with the key step using the inequality between the 2-norm and the 1-norm. See also the alternative proof provided by reviewer vtMw.

6. See answer to first Question of Reviewer MQLz.

We thank the reviewer for his careful review and positive comments about our paper. We address his main concerns and answer questions below.

Weaknesses

• see Answer to Q1/Q2 of Reviewer vtMw: our study show the clear statistical and geometrical benefits of using $d_{KT}$ distance over $MMD$, both theoretically and experimentally. Yet, the computational cost of using our distance is higher than MMD, so this reflects a tradeoff.

• empirical limitations: please see our answer to Q4-5-6 of Reviewer S4kf.

• particle flow: we will add a brief summary of the particle flow results in the main text to complement the details provided in the appendix B.3. Thank you for the suggestion.

Questions

• In practice, the assumptions on eigenvalue decay are not overly restrictive. There are known connections between empirical and population-level spectral decay, which make it possible to validate the assumptions through simulation. For example, even a uniform distribution over a cube exhibits polynomial decay when using the Gaussian kernel—despite the fact that the Wasserstein distance suffers from the curse of dimensionality in such settings. Similar behavior holds for other kernel-distribution pairs. For further discussion and concrete examples (including closed-form cases), we refer to Francis Bach’s blog series “Unraveling spectral properties of kernel matrices”, Parts I and II. We will add this discussion to the revised version of the manuscript.

• The Trace Distance corresponds to the case $p = 1$, while MMD with kernel $k^2$ is equivalent to the case $p = 2$. As $p$ increases, the space of discriminating functions in the IPM framework becomes smaller, while the associated convergence rates improve.

• One possible extension would be to learn the feature map using a neural network, as in the work of Mroueh et al. (2017), cited in the introduction. This could be done by training on a subset of the data or using an unsupervised approach. In that case, the RKHS would be finite-dimensional, and the kernel could be computed directly as $k(x, y) = \langle \varphi(x), \varphi(y) \rangle$, where $\varphi$ denotes the learned feature map. The difference kernel matrix and its $p$-Schatten norm could then be evaluated accordingly. However, it is important to note that the learned feature map must capture information relevant to both distributions simultaneously, which is not guaranteed if the training is performed using only one of them.

• In our work, we focused on MMD, the Wasserstein distance, and the Kernel Trace distance. In particular, Lemma 3.3 and Corollary 3.4 (for the Gaussian kernel) establish the inequality: $MMD_{k^2}(\mu, \nu) \leq d_{KT}(\mu, \nu) \leq \frac{2}{\sigma} W_{\|\cdot\|}(\mu, \nu).$ All three are instances of Integral Probability Metrics (IPMs). When choosing among them, one useful guideline is that moving higher in this hierarchy yields more discriminative power, but at the cost of slower statistical convergence and greater computational complexity.

We thank the reviewer for his constructive review, and discuss his concerns below.

1-3: We address these pitfalls of MMD more fully in Section 3.2 and Appendix B.1. Let us clarify our findings afterwards. Section 3.2 of the paper introduces the concept of normalized energy in the context of comparing probability measures via RKHS-based operators. We highlight a key distinction between our proposed Kernel Trace (KT) distance and the widely used Maximum Mean Discrepancy (MMD), that both a covariance mean embedding. The information about the distribution is contained in the eigenvalues (similar to probability mass) and eigenvectors (similar to distribution support) of the RKHS density operator. The MMD relies on the Schatten 2-norm and is sensitive to the internal "energy" $||\Sigma_\mu|| = \int\int k(x,y)^2 \mu(x) \mu(y)dx dy = \int_\mathcal{X} \int_\mathcal{Y} e^{-||x-y||^2/\sigma^2} \mu(x) \mu(y)dx dy$, (where $||\cdot||$ denotes the 2-norm here), which is itself related to the variance (or "spread") of distributions. It depends not only on the distribution but also on the kernel's bandwidth. Since our focus is on the distance between two embeddings—rather than representing a single embedding—we showed in Section 3.2 that embeddings with low energy can yield a low MMD value even when they differ significantly. This is because the bandwidth parameter cannot be simultaneously tuned to capture both scale and difference effectively. In contrast, the $d_{KT}$ distance uses the Schatten 1-norm, which equals one for all distribution (for all $\mu$, $|\Sigma_{\mu}| = 1$ where $|\cdot|$ denotes the 1-norm here) since we assumed the trace of RKHS density operators is 1. This normalization ensures that the KT distance focuses purely on the separation between distributions, not on their individual energy levels. Our empirical results (e.g., Figure 1) show that KT distance behaves better in this sense than MMD when comparing two fixed distributions while increasing the kernel bandwidths: $d_{KT}$ decreases as expected, while MMD is not monotone. In Figure 2, one can see that the $d_{KT}$ distance attains its theoretical maximal value (2), while MMD achieves a much smaller value (value 2 would be approachable by two far-away Dirac). In Figure 3, our distance drops below its maximum value of 2 only when the supports begin to significantly overlap (i.e. when the std deviation $s$ is of the order of the distance between the mean); in contrast, MMD decreases much earlier, as it naturally diminishes with increasing variance. Furthermore, as a second step, we demonstrate (Proposition 3.5) that for mixtures of distributions, the $d_{\mathrm{KT}}$ distance scales linearly with the component-wise distances, thereby preserving discriminability, whereas MMD tends to lose resolution (by a factor 2) due to the averaging effect inherent in its energy-based formulation. Generally, this section is intended to justify why $d_{KT}$ leads to a more stable and robust behavior, making it more reliable in scenarios where MMD’s performance deteriorates. We will highlight those clarifications in paper.

2: $k^2$ is just the squared function of the kernel function $k$ (whose general notation $k : \mathcal{X} \times \mathcal{X} \to \mathbb{R}$ as can be found as the very beginning of section 2), therefore $k^2 : \mathcal{X} \times \mathcal{X} \to \mathbb{R}, (x,y) \mapsto k(x,y)^2$ (positive definiteness is carried on by squaring). Notice that $\langle \varphi(x) \otimes \varphi(x), \varphi(y) \otimes \varphi(y) \rangle = \langle \varphi(x), \varphi(y) \rangle \langle \varphi(x), \varphi(y) \rangle = k(x,y)^2$.

4. We agree with the reviewer’s observation that, in statistical testing, the scale of a test statistic alone does not determine its effectiveness; rather, it should be evaluated in relation to the corresponding testing threshold under the null distribution. However, the utility of a distance metric is not limited to its discriminative power near the null—it also depends on its behavior for distributions that are moderately different, which is particularly relevant in applications such as particle flows or Approximate Bayesian Computation (ABC). We also refer to our final point, where we highlight the advantages of $d_{KT}$ over MMD in such settings. As a direction for future work, it would be valuable to conduct a theoretical and empirical comparison of the testing power of both metrics under common thresholds. However, we believe these questions involves substantially more work that we leave for a future study.

5. The paper by Mroueh et al., cited in the introduction, hinted at the potential of using this class of IPMs for generative adversarial networks. We acknowledge that it would be interesting to investigate our closed-form expression of the $d_{KT}$ distance leads to an improvement of their numerical results, while they were relying on an approximation of this distance. This may reveal further benefits of $d_{KT}$ in a high-dimensional setting. Our work serves as a first step in this direction, providing both theoretical foundations and empirical evidence to support the use of this distance.

6. Let us clarify the experimental setup and the results obtained for ABC. The Kernel Trace distance consistently achieves the lowest mean squared error (MSE), particularly when using a well-chosen value of $\varepsilon$. As we discussed, the fact that MMD accepts samples for very small $\varepsilon$ values—even when other distances reject them—is not necessarily an advantage. In such cases, acceptance under MMD may occur despite model misspecification, whereas rejection by $d_{KT}$ can serve as a useful indicator of such misspecification. We hope the following point will further address any concerns regarding the choice of $\varepsilon$.

To further demonstrate the advantage of using $d_{KT}$ over MMD in robust learning, we include an additional ABC experiment using linear regression on real data.

We consider normalized data $(X, Y) = \{(x_j, y_j)\}_{j=1}^n$, and aim to learn a parameter $\theta$ that minimizes the average squared bias, $(\theta^\top x_j - y_j)^2$. While ridge regression is a standard method for this task, we introduce an additional challenge by contaminating 10% of the training labels—shifting them by +10—to simulate label noise.

As in previous experiments, we apply ABC to sample $\theta_i \sim \mathcal{N}(0, 0.1 I_d)$, using a zero-centered prior with low variance, consistent with the normalization of the data. We split the data into 90% training and 10% testing, and use a Gaussian kernel with bandwidth equal to the data dimension $d$. For each of the 10 000 sampled $\theta_i$, we generate pseudo-labels $\tilde{Y} = \theta_i^\top X + Z$, where $Z \sim \mathcal{N}(0,1)$, i.e., $P_{\tilde{Y}|X} = \mathcal{N}(\theta_i^\top X, 1)$.

We then rank the sampled $\theta_i$ based on the distance between the generated distribution $(X, \tilde{Y})$ and the real data $(X, Y)$, using either MMD or $d_{KT}$. Instead of applying a hard threshold $\varepsilon$, we select the top 10% (i.e., the 1 000 smallest-distance samples) to avoid sensitivity to threshold choice—see reference [1] and Algorithm 3 in [2] for the quantile-based selection approach.

Below, we report the empirical average squared bias on the test data for the selected $\theta_i$, as well as for ridge regression. Since our prior has covariance $0.1 I_d$, the corresponding ridge penalty is $\lambda = 1 / 0.1 = 10$, though we also include other values of $\lambda$ for completeness.

As shown in the table, except for the first dataset, the Kernel Trace distance consistently outperforms MMD.

[1] Biau, G., Cérou, F., & Guyader, A. (2015). New insights into approximate Bayesian computation. In Annales de l'IHP Probabilités et statistiques [2] Robert, C. P. (2016). Approximate Bayesian computation: a survey on recent results

We hope our responses have addressed the reviewer’s concerns and that they will consider revising their score.

We thank the reviewer for his careful review and positive comments about our paper. We address his main concerns and answer questions below.

** Weaknesses **

• Thank you for your insightful comment regarding how to achieve different empirical convergence rates through projection-based function classes. We agree that such constructions—like the k-dimensional max-sliced 1-Wasserstein distance—can circumvent the curse of dimensionality to some extent by leveraging low-dimensional structure. We will incorporate this perspective into the revised manuscript.

That said, our focus in posing the original question was more on “natural” (non-compositional) function classes, such as those used in MMD and Kernel Trace Distance, which do not rely on explicit projections onto subspaces. These classes are typically characterized by covering or entropic numbers, and their complexity is measured directly in the ambient space. In this sense, our contribution offers a perspective that is somewhat orthogonal to the one you raised: the convergence behavior of our kernel-based distance may depend not only on the function class but also on properties of the distribution itself (e.g., the decay rate of eigenvalues), rather than solely on dimensionality or projection structure. We have rewritten this part in the introduction for better clarity.

• computational complexity: we acknowledge the greater computational complexity of our distance in comparison with MMD and believe reducing its computational cost is an interesting research direction for further work. Note yet that Sinkhorn's divergence scales as $\mathcal{O}(n^2\epsilon)$ where $n$ is the number of samples and $\epsilon$ is the entropic regularisation parameter, while recently introduced alternative kernel-based divergences to MMD also suffer from this $\mathcal{O}(n^3)$ (see Chazal et al 2024, which was presented last Neurips).

• Thank you for the improvement of factor 2 of Prop. 3.6.

• Thank you also for the spotting the typos. A completely separable space, also known as second-countable means that the topology has a countable basis.

** Questions **

• Q1: A way of motivating $d_{KT}$ from first principles is (see also answer to reviewer tydU) that it is part of a family of distances - IPM using $p$-Schatten norm- which includes the well established MMD, and it is the most "discriminative" inside this family.

• Q2: MMD is a suitable choice over $d_{KT}$ when it sufficiently discriminates between the given distributions, and when computational efficiency is a priority —such as when computing such distances must be run repeatedly or when working with a very large number of samples. As we wrote in the conclusion, one could investigate the use of the Nyström method (subsampling a lesser number of samples) to diminish the computation time of $d_{KT}$.

• Q3: Thank you for the suggestion—considering a centered version of the covariance is indeed an interesting direction. There are two natural approaches: the first, as in your equation, subtracts a constant term from each entry; the second subtracts the outer product of the mean (i.e., centering by removing the mean tensor from the covariance). Thanks to our new difference-kernel trick, the latter is now computationally feasible.

Implementing this second approach involves augmenting the difference kernel matrix by two additional rows and columns (corresponding to the means of each distribution). These new rows and columns are derived from averages of existing rows/columns, so the overall rank remains unchanged, simply adding two zero eigenvalues. Since the extended matrix is constructed from the original difference kernel, we expect its influence on the result to be limited—but it would be worthwhile to verify this empirically. We will add a discussion paragraph for future work mentioning this consideration.

• Q4: The paper [1] establishes lower bounds of order $O(n^{-1/2})$ for MMD when using radial kernels. By the inequality $\|\cdot\|_2 \leq \|\cdot\|_1$, this lower bound should, a fortiori, extend to the kernel trace distance. Moreover, it aligns with the fast convergence rates observed under exponential spectral decay, up to logarithmic factors. For other types of kernels, to the best of our knowledge, corresponding lower bounds for MMD have not yet been established.

[1] Tolstikhin, I. O., Sriperumbudur, B. K., & Schölkopf, B. (2016). Minimax estimation of maximum mean discrepancy with radial kernels. Neurips.

We thank the reviewer for his careful review and positive comments about our paper. We address his main concerns below.

** Questions **

• The 1-Schatten norm holds a special place among the $p$-Schatten norms for $p \geq 1$, as it is the largest in that family (since $\|\cdot\|_s \leq \|\cdot\|_p$ for $p \leq s$). This has important implications for Integral Probability Metrics (IPMs). Each $p$-Schatten norm admits a dual formulation via Hölder’s inequality, using the conjugate exponent $q$ such that $1/p + 1/q = 1$. In particular, when $p = 1$, the dual norm is the $\infty$-norm, which is the least restrictive and thus defines the largest possible set of discriminator functions in the IPM. In this sense, higher values of $p$ lead to weaker discriminative power, since the function class has the same structure as $\mathcal{F}_1$, but with the operator $U$ constrained in the $q$-norm rather than the $\infty$-norm. Furthermore, the 2-Schatten norm corresponds to MMD with kernel $k^2$. But since any positive definite kernel $k$ allows a square root $k'$ such that $k = (k')^2$, we can see MMD as a special case within this family. This leads to the conclusion that the KT distance is, in fact, the largest among this class of kernel-based IPMs, including MMD. The proof of Proposition 3.6 works indeed for all $p\geq 1$ (see l939 in the Appendix) because of the inequality between the norms.

• In Prop. 3.5, indeed, the orthogonality condition is $\Sigma_{\mu_1} \perp \Sigma_{\mu_2}, \Sigma_{\mu_1} \perp \Sigma_{\nu_2}, \Sigma_{\nu_1} \perp \Sigma_{\mu_2}, \Sigma_{\nu_1} \perp \Sigma_{\nu_2}$.

• Thank you for noticing the typos, we have corrected them. At line 328, this was indeed a typo — we intended to write “For the traditional likelihood” to indicate that it does not apply in this context. We apologize for the confusion. We hope this will clarify why standard Bayesian method (using Bayes'rule update) fails and robust ABC using our novel distance is necessary. $\phi$ was indeed $\varphi$ and we have removed the unnecessary subsection 4.1. We will also recall the fidelity with the Fuchs-van de Graaf's inequalities.

** Weaknesses **

Thank you for suggesting writing improvements. Below we clarify the message of Section 3.2 and ABC in Section 5.

• Section 3.2: This section of the paper introduces the concept of normalized energy in the context of comparing probability measures via RKHS-based operators. We highlight a key distinction between our proposed Kernel Trace (KT) distance and the widely used Maximum Mean Discrepancy (MMD), that both a covariance mean embedding. The information about the distribution is contained in the eigenvalues (similar to probability mass) and eigenvectors (similar to distribution support) of the RKHS density operator. The MMD relies on the Schatten 2-norm and is sensitive to the internal "energy" $||\Sigma_\mu|| = \int\int k(x,y)^2 \mu(x) \mu(y)dx dy = \int_\mathcal{X} \int_\mathcal{Y} e^{-||x-y||^2/\sigma^2} \mu(x) \mu(y)dx dy$, (where $||\cdot||$ denotes the 2-norm here), which is itself related to the variance (or "spread") of distributions. It depends not only on the distribution but also on the kernel's bandwidth. Since our focus is on the distance between two embeddings—rather than representing a single embedding—we showed in Section 3.2 that embeddings with low energy can yield a low MMD value even when they differ significantly. This is because the bandwidth parameter cannot be simultaneously tuned to capture both scale and difference effectively. In contrast, the $d_{KT}$ distance uses the Schatten 1-norm, which equals one for all distribution (for all $\mu$, $|\Sigma_{\mu}| = 1$ where $|\cdot|$ denotes the 1-norm here) since we assumed the trace of RKHS density operators is 1. This normalization ensures that the KT distance focuses purely on the separation between distributions, not on their individual energy levels. Our empirical results (e.g., Figure 1) show that KT distance behaves better in this sense than MMD when comparing two fixed distributions while increasing the kernel bandwidths: $d_{KT}$ decreases as expected, while MMD is not monotone. In Figure 2, one can see that the $d_{KT}$ distance attains its theoretical maximal value (2), while MMD achieves a much smaller value (value 2 would be approachable by two far-away Dirac). In Figure 3, our distance drops below its maximum value of 2 only when the supports begin to significantly overlap (i.e. when the std deviation $s$ is of the order of the distance between the mean); in contrast, MMD decreases much earlier, as it naturally diminishes with increasing variance. Furthermore, as a second step, we demonstrate (Proposition 3.5) that for mixtures of distributions, the $d_{\mathrm{KT}}$ distance scales linearly with the component-wise distances, thereby preserving discriminability, whereas MMD tends to lose resolution (by a factor 2) due to the averaging effect inherent in its energy-based formulation. Generally, this section is intended to justify why $d_{KT}$ leads to a more stable and robust behavior, making it more reliable in scenarios where MMD’s performance deteriorates. We will highlight those clarifications in paper.

-ABC in Section 5: Traditional Bayesian methods rely on computing the posterior distribution over models by updating a prior based on observed data. This can be computationally expensive, and a misspecified prior may lead to incorrect inference. As an alternative to Bayes' rule in such cases, one can use Approximate Bayesian Computation (ABC).

A key reference is Algorithm 1 in Appendix B.2. ABC is a Monte Carlo method in which models $p_{\theta_i}$ are sampled from the prior distribution, repeated for $i = 1, \dots, T$ iterations. Each sampled model is then evaluated for its fit to the observed data using a criterion such as a distance metric. The subset of models with the best evaluations is retained, forming an approximate posterior distribution over models.

In our setting, rather than using a likelihood, we compare distributions using a distance metric, and the threshold $\varepsilon$ controls the degree of model mismatch tolerated. This is particularly useful in scenarios with contaminated data, even when the model is otherwise correct. Because our distance metric tolerates a fraction $\varepsilon$ of mass to be mismatched (rather than, for instance, tolerating a shift in the mean), it offers greater robustness and efficiency—especially since the mean is not a robust statistic.

We are also expanding this section with additional ABC experiments; see our response to Reviewer S4kf at the end of the rebuttal for details.

We hope our responses have addressed the reviewer’s concerns and that they will consider revising their score.