Discriminative Estimation of Total Variation Distance: A Fidelity Auditor for Generative Data

We are grateful to the reviewer for the thoughtful and detailed feedback. We have made our best effort to address your concerns and present our point-by-point response for your questions and weaknesses that you highlight below.

For Weakness: We would like to share our understanding of this problem and highlight the motivation behind our paper. To our knowledge, any metric measuring the distributional difference between two datasets can be used to evaluate the quality of generative data, such as $f$-divergence, Wasserstein distance, and so on. However, there are several difficulties: (1) the high dimensionality of data and (2) unknown distribution forms. The first problem can be alleviated by employing an embedding model to obtain a low-dimensional representation of the data. For example, Inception V3 [1] takes $299\times 299\times 3$ images as input and outputs 1000-dimensional embeddings. To tackle the second challenge, a common approach is to assume the normality of embeddings, from which some metrics can have a closed-form formula, such as KL divergence and Wasserstein distance. Therefore, two natural questions arise

• Q1: Is it possible to estimate other distributional metrics without a closed-form formula under the Gaussian assumption? Since these metrics also provide a perspective of data fidelity.

• Q2: Is it possible to relax the Gaussian assumption?

To answer the above two questions, we consider the estimation of TV distance under the Gaussian assumption (Theorems 3.4 and 3.5). Later, we extend our results to the general exponential family (Theorem 3.7). Our theoretical results provide partial answers to these two questions. The proposed framework can be extended to other members of $f$-divergence, depending on the choice of loss function used for evaluation. In what follows, we summarize some connections between $f$-divergence metrics and surrogate loss functions. For example,

• Exponential loss and the Hellinger distance: $\min_{f}R_{\exp}(f)=\min_{f} \mathbb{E}(\exp(-f(X)Y))=1-2h^2(\mathbb{P},\mathbb{Q})$, where $X \sim \frac{\mathbb{P}(X)+\mathbb{Q}(X)}{2}$ and $h(\mathbb{P},\mathbb{Q})$ represents the Hellinger distance between the distributions $\mathbb{P}$ and $\mathbb{Q}$.

Conclusion As long as we can consistently estimate $\min_{f}R_{\exp}(f)$ or $\min_{f}R_{log}(f)$, it is achievable to estimate the associated divergence. Therefore, our current framework can be extended to other $f$-divergence metrics.

For Question 1: As discussed above, the key motivation for estimating the TV distance comes from our Q1 above. The TV distance can be used to evaluate the distributional difference between real and synthetic data. However, no closed-form solution is provided, even under the Gaussian assumption. Additionally, another motivation for estimating the TV distance lies in privacy auditing within the domain of differential privacy [2]. Specifically, total variation distance is a fundamental measure in privacy analysis, directly tied to the operational guarantees of differential privacy. Specifically: (1) Quantifying Worst-Case Privacy Leakage: In $(\epsilon, \delta)$-differential privacy, TV distance measures the allowable divergence between the distributions of outputs generated from two neighboring datasets, with $\delta$ representing the probability of deviation beyond the $\epsilon$-bound. By providing a lower bound on total variation, our tool identifies the minimum possible divergence, effectively quantifying the worst-case privacy leakage that cannot be avoided by any mechanism under these conditions. This is particularly relevant for applications such as generative data models, where synthetic data's resemblance to real data must be carefully balanced against strict privacy requirements.

For Question 2 We carefully reviewed the results in [3] and found that their analysis essentially provides a range for the TV distance between two Gaussians: $\frac{1}{200}\Delta \leq \text{TV}(N(\mu_1, \Sigma_1), N(\mu_2, \Sigma_2)) \leq \frac{1}{\sqrt{2}}\Delta$ where $\Delta$ is deterministic, given $\mu_1, \mu_2, \Sigma_1$, and $\Sigma_2$. In contrast, our proposed method provides a consistent estimation of the TV distance. We also thank the reviewer for bringing this related literature to our attention, which we have now included in our discussion.

References:

[1] Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J. and Wojna, Z., 2016. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2818-2826).

[2] Koskela, A. and Mohammadi, J., 2024. Black Box Differential Privacy Auditing Using Total Variation Distance. arXiv preprint arXiv:2406.04827.

[3] Arbas, J., Ashtiani, H. and Liaw, C., 2023, July. Polynomial time and private learning of unbounded gaussian mixture models. In International Conference on Machine Learning (pp. 1018-1040). PMLR.

We would like to express our gratitude to the reviewer for the insightful feedback. We have made every effort to address your concerns and present our point-by-point responses to the questions and weaknesses you highlighted below. All revisions are highlighted in Blue in the revised manucsript.

For Weakness 1: We would like to share our understanding of this problem and highlight the motivation behind our paper. In the literature, distributional-difference metrics are used to evaluate the fidelity of generative data, such as $f$-divergence, Wasserstein distance, and so on. However, there are two main difficulties: (1) the high dimensionality of data and (2) unknown distribution forms. The first problem can be alleviated by employing an embedding model to obtain a low-dimensional representation of the data. For example, Inception V3 [1] takes $299\times 299\times 3$ images as input and outputs 1000-dimensional embeddings. To tackle the second challenge, a common approach is to assume the normality of embeddings, from which some metrics can have a closed-form formula, such as KL divergence and Wasserstein distance. However, two natural questions arise

• Q1: Is it possible to estimate other distributional metrics without a closed-form formula under the Gaussian assumption? Since these metrics also provide a perspective of data fidelity.

• Q2: Is it possible to relax the Gaussian assumption?

To answer the above two questions, we consider the estimation of TV distance under the Gaussian assumption (Theorem 3.4). Later, we extend our results to the general exponential family (Theorem 3.6) for removing the Gaussian assumption. Our results provide partial answers to these two questions.

For Weakness 2: We thank the reviewer for pointing out the shortcomings in our literature review. We have now included the methods you suggested in the updated version of our review.

For Weakness 3 and Question 2: We thank the reviewer for pointing out the possible improvement in our method. Following your suggestion, we have specified those constants in Theorem 3.5. We specify the values of $C_0$ and $\gamma$ when $\mathbb{P}$ and $\mathbb{Q}$ are multivariate Gaussian distributions with identical covariance matrices. In this case, the result becomes

$$\mathbb{E}_{\mathcal{D}} \Big( \widehat{\mathrm{TV}}(\mathbb{P}, \mathbb{Q})-\mathrm{TV}(\mathbb{P}, \mathbb{Q}) \Big) \lesssim \Delta^{-\frac{1}{3}} \Big( \frac{d\log n}{2n} \Big)^{\frac{2}{3}}$$

where $\Delta = ( \mathbf{\mu}_1-\mathbf{\mu}_2)^T \mathbf{\Sigma}( \mathbf{\mu}_1-\mathbf{\mu}_2)$. In this result, we show that $C_0 \asymp 1/ \Delta$ and $\gamma=1$. Our findings illustrate that the proposed discriminative estimation method achieves a rapid convergence rate of $O\Big(\Delta^{-1/3}n^{-\frac{2}{3}}\Big)$, accompanied by a logarithmic factor. Notably, as $\Delta$ tends towards infinity ($\mathbf{\mu}_1$ becomes more different from $\mathbf{\mu}_2$), the convergence rate accelerates, consistent with our second observation mentioned earlier.

For Weakness 5 and Question 1: We thank the reviewer for the suggestion, which is also suggested by Reviewer jVgz. Following your suggestion, we further included the CIFAR10 dataset in our real application. Our experimental result shows that the proposed method ensures consistent fidelity ranking of image data.

More details about the implementations can be found in the revised manuscript.

Thank you very much for the time and effort that you have put into reviewing our paper. We greatly appreciate your detailed comments and constructive suggestions for improving the quality of this paper. In what follows, we provide point-by-point response to the question and weaknesses that you highlight. All revisions are highlighted in Blue in the revised manuscript.

For weakness 1: Basically, the proposed framework can be easily extended to other members of $f$-divergence, depending on the choice of loss function used for evaluation. In what follows, we summarize some connections between $f$-divergence metrics and surrogate loss functions:

• Exponential loss and the Hellinger distance: $\min_{f}R_{\exp}(f)=\min_{f} \mathbb{E}(\exp(-f(X)Y))=1-2h^2(\mathbb{P},\mathbb{Q})$, where $X \sim \frac{\mathbb{P}(X)+\mathbb{Q}(X)}{2}$ and $h(\mathbb{P},\mathbb{Q})$ represents the Hellinger distance between the distributions $\mathbb{P}$ and $\mathbb{Q}$.

• Logistic loss and Jensen-Shannon divergence: $\min_{f}R_{log}(f)=\min_{f} \mathbb{E}(\log(1+\exp(-f(X)Y)))=\log 2 - 2JS(\mathbb{P},\mathbb{Q})$, where $X \sim \frac{\mathbb{P}(X)+\mathbb{Q}(X)}{2}$ and $JS(\mathbb{P},\mathbb{Q})$ represents the Jensen-Shannon divergence between the distributions $\mathbb{P}$ and $\mathbb{Q}$.

The conclusion is that as long as we can consistently estimate $\min_{f}R_{\exp}(f)$ or $\min_{f}R_{log}(f)$, it is achievable to estimate the associated divergence. Therefore, our current framework can be extended to other $f$-divergence metrics. More details about deriving these results can be found in [1]. Moreover, we agree that the Wasserstein distance is a practical metric under the Gaussian assumption. However, the main purpose of this paper is to provide a practical alternative for evaluating the fidelity of generative data. Furthermore, our results can be extended to other distributions belonging to the exponential family, as summarized in Table 1 of the manuscript.

For weakness 2 and Question Following your suggestion, we have included CIFAR10 in our experiment for verifying the effectiveness of the proposed method. Our experimental result shows that the proposed method ensures consistent fidelity ranking of image data.

For weakness 3 In this paper, our main focus is generalizing the Gaussian assumption to the exponential family. In Theorem 3.7, we specify the explicit form of optimal classifier for other exponential family. This result implies that the corresponding hypothesis class in our framework should be $\mathcal{H}=\{f(\mathbf{x})=\mathbf{\beta}^T \psi(\mathbf{x})\}$ with $\psi(\mathbf{x})=(T_1(\mathbf{x}),T_2(\mathbf{x}))$, where $T_1(\mathbf{x})$ and $T_2(\mathbf{x})$ are sufficient statistics of $\mathbb{P}$ and $\mathbb{Q}$. We also provide more explicit resuls for 1-dimensional case in Table 1 of our paper

For weakness 4 We thank the reviewer for carefully examining the claims in our paper. We apologize for the lack of clarity in our initial submission. $\text{TV}(\mathbb{P}, \mathbb{Q}) \ge 1-2R(\widehat{f}) \triangleq \widehat{\text{TV}}(\mathbb{P}, \mathbb{Q})$ Here, when $\widehat{f}=f^\star$ (The Bayes classifier), we have $\widehat{\text{TV}}(\mathbb{P}, \mathbb{Q})=\text{TV}(\mathbb{P}, \mathbb{Q})$. Therefore, our claim is that if the Bayes classifier fails to discriminate between real and synthetic data, their underlying distributions should be the same. Following your suggestion, we have rewritten this claim to avoid any misunderstanding.

References:

[1] XuanLong Nguyen. Martin J. Wainwright. Michael I. Jordan. "On surrogate loss functions and f-divergences." Ann. Statist. 37 (2) 876 - 904