A Kernel Distribution Closeness Testing

Thank you for reading our paper and for your insightful comments. We provide more explanations of the definition of our NAMMD below.

[Q1] The author does not explain why the definition of NAMMD is formatted as Equation (1). Although ..., the fundamental motivation for designing NAMMD is not clearly described. Why the definition of NAMMD is formatted as Equation (1) and sufficient to achieve the desired goal? This point lacks explanation. After defining NAMMD, the authors do not provide a corresponding justification.

[A1] Thank your for pointing out this concern about our fundamental motivation. We will add a Remark immediately after the definition of NAMMD in Equation (1), providing the detailed explanation outlined as follows.

In our NAMMD, we aims to capture differences between two distributions based on their kernel mean embeddings (i.e. $\mu_P$ and $\mu_Q$), leveraging the property that a kernel mean embedding uniquely represents a probability distribution and capture unique characteristics to support efficient distribution comparisons [Sriperumbudur et al., 2011]. To measure the difference, a natural way is to calculate the distancen between two mean embeddings (i.e. $||\mu_P-\mu_Q||^2_H$), which is the MMD distance. However, we find that the MMD can yield same values for many pairs of distributions (i.e. $P$ and $Q$) that have different norms (i.e. $||\mu_P||^2_H$ and $||\mu_Q||^2_H$), which actually indicates different levels of closeness. For example, in Figure 1a and 1b, distribution pair ($P_1,Q_1$) shares the same MMD distance as ($P_2, Q_2$) but has different norms. Larger norms indicate more tightly concentrated distributions, leading to smaller standard deviations and $p$-values of the empirical MMD estimator as shown in Figure 1c, even with a constance MMD value $0.15$. Notably, smaller $p$-values indicate more significant difference and less closeness between distributions. Hence, the MMD fails to distinguish the closeness levels for the distribution pairs in Figures 1a and 1b. To mitigate this issue, we scale the MMD distance by incorporating the norms information $4K-||\mu_P||^2_H-||\mu_Q||^2_H$. It is evident that the NAMMD value scaled up as norms ($||\mu_P||^2_H+||\mu_Q||^2_H$) increase, which aligns with the fact that larger norms indicates larger distance between distributions (when MMD value remains constant). Figure 1c and 1d demonstrate that our NAMMD exhibits a stronger correlation with the $p$-value (i.e. the level of closeness), while the MMD value is held constant.

We further demonstrate the advantages of our NAMMD over the original MMD in Theorems 9 and 11, where the improvement in test power is achieved through scaling with $4K - ||\mu_P||^2_H - ||\mu_Q||^2_H$.

[Q2] As the author stated, MMD has an inherent issue in closeness testing: "the same MMD value may reflect different levels of closeness between distributions, which makes it less informative." Why doesn't the author redesign a kernel-based closeness testing method? Such approach seems to be able to circumvent the inherent problems of MMD and further address the closeness testing of high-dimensional data.

[A2] Thanks for pointing out this interesting Quetion. The key issue identified in this paper (i.e. the "less informative" of MMD) arises because MMD yields same values for many pairs of distributions that have different norms of mean embeddings (which results in different levels of closeness). By incorporating norm of mean embeddings in our NAMMD, we effectively mitigate this issue. In designing a new kernel-based closeness testing method, the mean embedding of a distribution is crucial, as it is injective for characteristic kernels, ensuring that different distributions have distinct embeddings and capturing unique characteristics to support efficient comparison for distributions [Sriperumbudur et al., 2011]. The MMD directly measures the Euclidean-like distance between mean embeddings, and its simplicity and well-established theoretical properties make it a reliable choice for various applications [Gretton et al., 2012]. Hence, our NAMMD builds on the importance of mean embeddings and strengths of MMD, enhancing the statistic's effectiveness in distribution closeness testing by incorporating additional information from the mean embeddings.

We genuinely appreciate your valuable insights. While we have made efforts to address your concerns, we acknowledge that there may still be aspects requiring further clarification. We would be delighted to engage in further discussions to ensure your concerns are fully addressed.

Thank you for your valuable comments. We have submitted our response to address your concerns and answer your questions. We would greatly appreciate it if you could let us know whether our response has resolved your concerns, and we look forward to further discussing this with you.

Dear Reviewer 8GF1,

Thank you for taking the time and effort to review our manuscript.

We have carefully addressed all your comments and prepared detailed responses. Could you kindly review them at your earliest convenience?

We hope our responses have satisfactorily addressed your key concerns. If anything remains unclear or requires further clarification, please do not hesitate to let us know, and we will address it promptly.

We look forward to your feedback.

Best regards,

Authors of Submission 6378

Dear Reviewer 8GF1,

Thank you again for your time and effort in reviewing our paper.

We have carefully addressed all your comments and prepared detailed responses. Could you kindly review them at your earliest convenience?

As the rebuttal period is coming to a close, we wanted to check if there are any remaining concerns preventing you from adjusting your score. If there is any additional clarification we can provide, please do not hesitate to let us know and, we will address it promptly.

Best regards,

Authors of Submission 6378

Thank you for reading our paper and for your insightful comments. We provide further theoretical analysis of the variance estimator and details of the NAMMDFuse below.

[Q1] The threshold plays a key role in hypothesis testing. .... Under the case where the authors estimate it based on the variance of the asymptotic distribution, ... using the empirical variance estimator $\sigma_{X,Y}$. Can the authors provide more theoretical analysis regarding this estimator...?

[A1] Thank you for your concern. We present that for our empirical variance estimator $\sigma_{X,Y}$ with sample size $m$, the following holds:

Proof Sketch:
For simplicity, we denote:

and

Hence, we establish the equivalence: $\sigma^2_{X,Y} =\hat{A}^2/\hat{B}^2$ and $\sigma^2_{P,Q}=A^2/B^2$. Building on this, we further investigate that:

Breaking it down:

The second equality relies on the unbiased variance estimator of the U-statistic, i.e., $\hat{A}$. We further have the following inequality:

where $C>0$ is a constant that ensures $\frac{\hat{A}^2(B+\hat{B})}{\hat{B}^2B^2} \leq C$, and it exists since the kernel is bounded.

Based on the large deviation bound for $B$, we have:

Integrating this probability:

Using substitution $u = mt^2/4K^2$:

[Q2] Can the authors elaborate more on how to modify NAMMD in the fusing statistics approach?

[A2] Thanks for pointing our this quesiton. We will include more details about NAMMDFuse in our revision. Following the fusing statistics approach of [1], we introduce the NAMMDFuse statistic through exponentiation of NAMMD with samples $X$ and $Y$ as follows

where $\lambda>0$ and $\widehat{N}(X,Y)=\frac{1}{m(m-1)}\sum_{i\neq j}^m \kappa(\mathbf{x}_i,\mathbf{x}_j)^2+\kappa(\mathbf{y}_i,\mathbf{y}_j)^2$ is permutation invariant. $\pi(\langle X,Y\rangle)$ is the prior distribution on the kernel space $\mathcal{K}$. In experiments, we set the prior distribution $\pi(\langle X,Y\rangle)$ and the kernel space $\mathcal{K}$ to be the same for MMDFuse.

[1] Biggs, F., Schrab, A., & Gretton, A. (2024). MMD-FUSE: Learning and combining kernels for two-sample testing without data splitting. Advances in Neural Information Processing Systems, 36.

We genuinely value your insights and would be delighted to engage in further discussions with you regarding your concern.

Thank you for your thoughtful comments. We have submitted our response to address your concerns and questions. We would greatly appreciate it if you could confirm whether our response resolves your concerns. We look forward to your feedback and further discussion.

Dear Reviewer u8Wh,

Thank you for taking the time and effort to review our manuscript.

We have carefully addressed all your comments and prepared detailed responses. Could you kindly review them at your earliest convenience?

We hope our responses have satisfactorily addressed your key concerns. If anything remains unclear or requires further clarification, please do not hesitate to let us know, and we will address it promptly.

We look forward to your feedback.

Best regards,

Authors of Submission 6378

Dear Reviewer u8Wh,

Thank you again for your time and effort in reviewing our paper.

We have carefully addressed all your comments and prepared detailed responses. Could you kindly review them at your earliest convenience?

As the rebuttal period is coming to a close, we wanted to check if there are any remaining concerns preventing you from adjusting your score. If there is any additional clarification we can provide, please do not hesitate to let us know and, we will address it promptly.

Best regards,

Authors of Submission 6378

Dear Reviewer u8Wh,

Thank you again for your time and effort in reviewing our paper.

We have carefully addressed all your comments and prepared detailed responses. Could you kindly review them at your earliest convenience?

As the discussion period is coming to a close, we wanted to check if there are any remaining concerns preventing you from adjusting your score. If there is any additional clarification we can provide, please do not hesitate to let us know, and we will address it promptly.

Best regards,

Authors of Submission 6378

Dear Reviewer u8Wh,

Thank you again for your time and effort in reviewing our paper.

We have carefully addressed all your comments and provided detailed responses. We have also revised our paper based on your suggestions. Could you kindly review them at your earliest convenience?

As the discussion period is coming to a close, we wanted to check if there are any remaining concerns preventing you from adjusting your score. If there is any additional clarification we can provide, please do not hesitate to let us know, and we will address it promptly.

Best regards,

Authors of Submission 6378

Dear Reviewer u8Wh,

Thank you again for your time and effort in reviewing our paper.

We have carefully addressed all your comments and provided detailed responses. We have also revised our paper based on your suggestions. Could you kindly review them at your earliest convenience?

As the discussion period is coming to a close, we wanted to check if there are any remaining concerns preventing you from adjusting your score. If there is any additional clarification we can provide, please do not hesitate to let us know, and we will address it promptly.

Best regards,

Authors of Submission 6378

Dear Reviewer u8Wh,

Thank you again for your valuable time and effort in reviewing our paper.

With the discussion period ending in less than 24 hours, we wanted to ensure all your concerns have been addressed and check if anything remains that might prevent you from updating your score. If there are any additional clarifications or adjustments you would like us to make, please let us know, and we will address them promptly.

Thank you once again for your consideration.

Best regards, Authors of Submission 6378

Thank you for reading our paper and for your insightful comments. We answer your questions and concerns below.

[Q1] The issue identified by this study may ... related to kernel selection ... The ... configuration of kernel hyperparameters may be related to the issue identified in this study. Some research has addressed issues related to kernel selection and its hyperparameter configuration ([1], [2], and [3]).

[A1] Thank you for pointing out this question. The selection of the kernel and the configuration of its parameters significantly impact the test results, which we did not discuss thoroughly in our original paper. Specifically, with selected kernels (e.g. those in [1], [2], and [3]), many distribution pairs can still exhibit the same MMD value but have different kernel norms, which actually indicates different closeness levels. In this paper, our NAMMD method leverages the norm information of distributions to mitigate the issue, which is compatible with existing kernel selection approaches (e.g., NAMMDFuse in the experiments shown in Fig.2, which is based on NAMMD and selected kernels from [1]; The kernel selection for our NAMMD provided in Appendix C.1 following Sutherland et al., 2017; Liu et al., 2020)). In Theorem 9 and 11, we prove that our NAMMD achieves better performance than MMD, when the same kernel is used for both distances. Naturally, if the optimal kernels are selected for MMD and NAMMD respectively, our NAMMD can still achieves better performance. In our current experiments, we use the same kernel for both MMD and NAMMD. Additional experiments exploring different optimal kernels for MMD and NAMMD will be conducted and included in the coming days.

[Q2] Lines 475-476: Does “$||\mathbf{z}_0, \mathbf{z}_1||^2 \geq ||\mathbf{z}_0, \mathbf{z}_2||^2 \geq \cdots \geq ||\mathbf{z}_0, \mathbf{z}_9||^2$” mean $||\mathbf{z}_0 - \mathbf{z}_1||^2 \geq ||\mathbf{z}_0 - \mathbf{z}_2||^2 \geq \cdots \geq ||\mathbf{z}_0 - \mathbf{z}_9||^2$?

[A2] Sorry for the typo, it should be the $||\mathbf{z}_0 - \mathbf{z}_1||^2 \geq ||\mathbf{z}_0 - \mathbf{z}_2||^2 \geq \cdots \geq ||\mathbf{z}_0 - \mathbf{z}_9||^2$.

[Q3] In lines 106-107, ... adjust the magnitude of the norm by modifying ... $\gamma$ of the Gaussian kernels. Specifically, .....

• ... did you observe any significant changes in test results by manually adjusting kernel hyperparameters, such as those of the Gaussian kernels?
• For existing MMD methods, can the selection of kernel parameters help mitigate the issues identified in this study? If so, what are the strengths and weaknesses of your approach compared to other existing MMD DCT methods that adjust kernel parameters ([1], [2], and [3])?

[A3] Thanks for your concern. It is possible to adjust the norms by modifying hyperparameter $\gamma$.

• Selecting kernel parameters is essential for capturing differences between distributions and significantly impacts test results. In our paper, we can select the kernel by maximizing the probability of correctly recognizing different distributions. This is grounded in maximizing the test power estimator from asymptotic distribution given in Theorem 2 and has been well studied in previous approaches (Sutherland et al., 2017; Liu et al., 2020). Further details are provided in Appendix C.1. We will include additional discussion and emphasize this in our revision.
• The selection of kernels can help mitigate the issues over specific distribution pairs. However, since the selected kernel is consistently used to evaluate the closeness of diverse and multiple distribution pairs for fair comparisons, the issue persists, particularly when testing distribution pairs not involved in the kernel selection process. By incorporating the norm information of distributions, our method effectively mitigates the issue in the general case and remains fully compatible with existing kernel selection approaches.

[Q4] Does the issue observed in this study, ... also occur for the MMD DCT when using the following two types of kernels?

• Unbounded kernels: ...
• Kernels with a positive limit at infinity: ...

[A4] Thanks for pointing out these interesting kernels. We will expand our discussion on kernel selection and demonstrate that the issue identified in this paper is a general problem across various kernel types and hyperparameter settings. Specifically, for the two types of kernels mentioned in the question, we provide examples to illustrate that the issue (i.e., the phenomenon where distributions with different norms can yield the same MMD) also occurs with the MMD DCT.

• Unbounded kernels: For polynomial kernels of the form

We define $P_1=\\{\frac{1}{4},\frac{3}{4}\\}$ and $Q_1=\\{\frac{1}{2},\frac{1}{2}\\}$ over vector domains $\\{(\sqrt{c},...,0),(-\sqrt{c},...,0)\\}$, respectively. Furthermore, we define $P_2=\\{\frac{3}{4},\frac{1}{4}\\}$ and $Q_2=\\{1,0\\}$ over domains $\\{(\sqrt{c},...,0),(-\sqrt{c},...,0)\\}$. It is evident that

with different norms for distributions pairs $||P1||_H^2 + ||Q_1||_H^2=\frac{9}{8}(2c)^d$, and $||P_2||_H^2 + ||Q_2||_H^2=\frac{13}{8}(2c)^d$. Specifically, we have $||P_1||_H^2=\frac{5}{8}(2c)^d$, $||Q_1||_H^2=\frac{1}{2}(2c)^d$, $||P_2||_H^2=\frac{5}{8}(2c)^d$ and $||Q_2||_H^2=(2c)^d$.

In a similar manner, for matrix products kernels of the form

and denote by $M_{11}$ the element in the first row and first column of the matrix $M$. We define $P_1=\\{\frac{1}{4},\frac{3}{4}\\}$ and $Q_1=\\{\frac{1}{2},\frac{1}{2}\\}$ over vector domains $\\{(\sqrt{c/M_{11}},...,0),(-\sqrt{c/M_{11}},...,0)\\}$, respectively. Furthermore, we define $P_2=\\{\frac{3}{4},\frac{1}{4}\\}$ and $Q_2=\{1,0\}$ over domains $\\{(\sqrt{c/M_{11}},...,0),(-\sqrt{c/M_{11}},...,0)\\}$. We can obtain the same results as discussed for polynomial kernels.

• Kernels with a positive limit at infinity: Using the given example kernel as $\kappa(\mathbf{x},\mathbf{x}')=\exp(-\frac{||\mathbf{x}-\mathbf{x}'||^2}{2\gamma})$ when $||\mathbf{x}-\mathbf{x}'||_{\infty}<K$, and otherwise $\kappa(\mathbf{x},\mathbf{x}')$ with positive constants $K$ and $c$. We define $P_1=\\{\frac{1}{4},\frac{3}{4}\\}$ and $Q_1=\\{\frac{3}{4},\frac{1}{4}\\}$ over vector domains $\\{(K,...,0),(4K,...,0)\\}$, respectively. Furthermore, we define $P_2=\\{\frac{1}{2},\frac{1}{2}\\}$ and $Q_2=\\{1,0\\}$ over domains $\\{(K,...,0),(4K,...,0)\\}$. It is evident that $$\textnormal{MMD}(P_1,Q_1)=\textnormal{MMD}(P_2,Q_2)=\frac{1}{2}(1-c)\ ,$$ with different norms for distributions pairs $||P_1||_H^2 + ||Q_1||_H^2=\frac{5+3c}{4}$, and $||P_2||_H^2 + ||Q_2||_H^2=\frac{3+c}{2}$. Specifically, we have $||P_1||_H^2=\frac{5+3c}{8}$, $||Q_1||_H^2=\frac{5+3c}{8}$, $||P_2||_H^2=\frac{1+c}{2}$ and $||Q_2||_H^2=1$.

[Q5] Background for Questions 3 (i.e. [Q4] Questions Regarding Kernel Selection).

[A5] We will add the background for kernel selection. A brief overview is as follows:

Characteristic kernels are essential in measuring distribution differences because their kernel mean embedding uniquely represents a probability distribution, capturing all its relevant information for comparison [Sriperumbudur et al., 2011]. Kernel selection is crucial in two-sample testing, helping to choose the best characteristic kernels for detecting differences in a specific distribution pair. Some methods select kernels supervisedly using held-out data [Sutherland et al., 2017], while others use unsupervised approaches like the median heuristic [Gretton et al., 2012] or adaptively select multiple kernels [Schrab et al., 2022; Biggs et al., 2023]. Selected kernels can effectively capture differences between distributions using MMD but lack norm information, leading to challenges in comparing closeness across diverse distribution pairs, regardless of kernel type or hyperparameter settings.

[Q6] In lines 084–102 and Figure 1, a kernel has been used such that $\lim_{||\mathbf{x}-\mathbf{x}'||_{\infty}\rightarrow\infty}=0$. This phenomenon could arise when a kernel fails to effectively measure similarity between distant data points. I am concerned that this issue might result from choosing a kernel that cannot capture similarity for data points beyond a certain distance.

[A6] Thanks for pointing out this concern. We use the commonly used Gaussian kernel as an example for illustration in Fig.1; however, the issue also arises when using more complex kernels, such as deep kernels [Liu et al., 2020](kernels based on neural network representations). The kernels used in our paper are all characteristic kernels, which ensure an injective mapping between mean embeddings and distributions, i.e., $P\rightarrow\mu_P$. This guarantees that different distributions have distinct embeddings, capturing unique characteristics to support efficient distribution comparison. Yet, the phenomenon still arises due to the lack of norm information of distributions in the original MMD, which we address by incorporating norm information in our NAMMD. We will add more discussion and highlight this in our revision.

We deeply appreciate your insights and would welcome the opportunity to discuss any further concerns you may have.

Thank you for your thoughtful reviews. We have carefully addressed your concerns and questions in our response. We would appreciate it if you could let us know whether our response has adequately resolved your concerns. We look forward to continuing the discussion.

[A1_exp] Dear reviewer, we add the experiments for [A1] as follows:

We conduct experiments to compare our NAMMD test and original MMD test in distribution closeness testing with optimal kernels. Following Definition 10, we perform the distribution closeness testing on two pairs of distributions: $P_1$ and $Q_1$, and $P_2$ and $Q_2$, where $\textnormal{NAMMD}(P_1,Q_1)=\epsilon$ and $\textnormal{NAMMD}(Q_2,P_2)=\epsilon+0.01$. For comparison, we set $\epsilon\in\\{0.1,0.3,0.5,0.7\\}$ and conduct experiments to assess if the distance between $P_2$ and $Q_2$ is larger than that between $P_1$ and $Q_1$. To construct distributions, we draw two sets of 500 elements from dataset, denoted as $Z$ and $Z'$. Let $P_1$ and $Q_1$ be uniform distributions over $Z$ and $Z'$. We then optimize $Z$ and $Z'$ by gradient method to ensure that $\textnormal{NAMMD}(P_1,Q_1)=\epsilon$ (Appendix D.2). It is straightforward to calculate $\textnormal{MMD}(P_1,Q_1)$ and to construct $P_2$ and $Q_2$ in a similar manner. For blob dataset, the test powers with optimal kernels are given as follows

The Table shows that our NAMMD achieves higher test power than MMD in distribution closeness testing with optimal kernels. We provide similar results for other four benchmark datasets in the following parts.

For higgs dataset, the test powers with optimal kernels are given as follows

For hdgm dataset, the test powers with optimal kernels are given as follows

For mnist dataset, the test powers with optimal kernels are given as follows

For cifar10 dataset, the test powers with optimal kernels are given as follows

Dear Reviewer TXYq,

Thank you for your response. We completely understand that you need time to gather your thoughts and provide feedback. Please feel free to take the time you need—we truly value your insights and look forward to receiving your response whenever you're ready.

If there is anything we can assist with or clarify further in the meantime, please do not hesitate to let us know.

Best regards, Authors of Submission 6378

[A1] Thank you for pointing out this question. The selection of ...

[A3] Thanks for your concern. It is possible to adjust the norms by modifying hyperparameter ...

Thank you for your detailed responses to the comments on your manuscript. I would also like to apologize for the delay in providing my feedback.

I agree that there are cases where kernel selection cannot be performed. However, the problem setting in your research appears to be ambiguous on this point. It might be better to clarify whether you are considering a scenario where kernel parameter selection is not performed, or if you allow situations where kernel parameter tuning is permitted.

Additionally, in the former case, it may be more appropriate to demonstrate the effectiveness of the proposed method through comparisons, such as minimax evaluations with respect to a kernel parameter set, rather than comparing estimation accuracies for specific parameters (Theorems 9 and 11).

[Q6] Thanks for pointing out this concern. We use the commonly used Gaussian kernel ...

Thank you for your detailed responses to the comments on your manuscript.

I disagree with the following point:

Yet, the phenomenon still arises due to the lack of norm information of distributions in the original MMD, which we address by incorporating norm information in our NAMMD. We will add more discussion and highlight this in our revision.

This point is difficult to agree with for the following two reasons:

1. The phenomenon of “lack of norm information in distributions” does not occur for characteristic kernels with optimal kernel parameters.
2. Even when the kernel parameters are not optimized, the results of your numerical experiments (Figure 1 (d)) may vary significantly depending on the choice of kernel. In particular, based on the definition of the MMD (line 139), I anticipate the MMD may increase in accordance with increases in the data variance (“Norms of Distributions”) in the case of "Unbounded kernels: For polynomial kernels of the form".

Dear Reviewer TXYq,

Thank you for taking the time and effort to review our manuscript.

We have carefully addressed all your comments and prepared detailed responses. Could you kindly review them at your earliest convenience?

We hope our responses have satisfactorily addressed your key concerns. If anything remains unclear or requires further clarification, please do not hesitate to let us know, and we will address it promptly.

We look forward to your feedback.

Best regards,

Authors of Submission 6378

Thank you very much for your detailed and thoughtful responses to my previous comments. I deeply appreciate the effort you have put into addressing the points I raised.

I would also like to sincerely apologize for the delay in providing my reply. I appreciate your patience in awaiting my feedback.

Below, I have summarized my thoughts and observations based on your responses. I would be grateful if you could share your opinions or any additional insights on these points. Please note that I plan to provide a specific feedback on each of your responses later. For now, I would like to focus on presenting a comprehensive summary of my concerns and suggestions.

The issue addressed in this study ("MMD value is less informative when measuring the closeness between two distributions, i.e., MMD value can be the same for many pairs of distributions that have different norms in the RKHS") mainly arises due to the selection of the kernel parameter used in the norm. I believe that the following four revisions focusing on this are necessary.

However, I am concerned that these changes are so significant that they may require a fundamental re-examination of the study. Could you please let me know your opinions on these points?

1. The need to accurately rewrite the description of the issue addressed in this study:

• The statement "MMD value is less informative when measuring the closeness between two distributions, i.e., MMD value can be the same for many pairs of distributions that have different norms in the RKHS" refers to the case when the kernel parameter is fixed. I believe that this issue does not occur when an appropriate kernel parameter is optimally selected with a characteristic kernel.

• Additionally, the assertion "MMD value can be the same for many pairs of distributions that have different norms in the RKHS" may not hold for kernels other than the Gaussian kernel (see Supplement Note 1 below).

2. Necessity to rewrite the main body of the paper to reflect point 1:

• Based on point 1, the discussion regarding kernel parameter selection should not be treated as merely supplementary. I believe that a comprehensive revision of the manuscript, including the introduction, is necessary.

3. Conducting numerical experiments considering point 1:

• Each numerical experiment should involve the selection of kernel parameters.

• Furthermore, I anticipate that the results of Figure 1 (c) and (d) would differ significantly if kernel parameter optimization is performed (see Supplement Note 2 below).

4. Addition or modification of theoretical results in response to point 1:

• The main result of this study might be more appropriately characterized as a "Proposal of a robust estimation method with respect to kernel parameter selection". The following theoretical aspects need to be described more rigorously to clarify the properties of the proposed method.

  • What effects can be expected from the proposed method? (e.g., the variance of the estimator remains below a certain level)

  • For which kernels can these effects be expected? (e.g., for characteristic kernels satisfying $\lim\_{\|\mathbf{x}-\mathbf{x}'\|\_{\infty} \rightarrow \infty} \kappa(\mathbf{x}, \mathbf{x}') = 0$)

Supplement Note 1:

This study proposes a method considering a kernel such as where $\operatorname{Var}(P, \kappa) = 1 - \\| \mu_P \\|^2\_{\mathcal{H}\_\kappa}$ and $\operatorname{Var}(P, \kappa) = 1 - \\| \mu_P \\|^2\_{\mathcal{H}_\kappa}$ (in line 90). I believe that the proposed method can achieve robustness in the MMD estimation (the variance of the estimator remains below a certain level), when there exists such a relationship between the increase in data variance and the decrease in embedding norm. On the other hand, for kernels not like this one (e.g., polynomial kernels on a compact support, where the embedding norm increases as the data variance increases), it is anticipated that the proposed method may not work well.

Supplement Note 2:

Instead of fixing the kernel parameters for these graphs, the results should be those where the kernel parameters are tuned for the variance of the data in each case (i.e., "Norms of Distributions"). I anticipate that the situation would change significantly in both graphs.

Thank you for your prompt and thoughtful response to my comments. I appreciate the effort you have made to address the concerns I raised in such a short time.

I believe that the approach you have considered completely addresses the concerns I raised.

In particular, I found that the following points are not only effective in clarifying the contributions of your research but also indicate directions for future research by identifying the two distinct issues, TST and DCT, in distribution closeness testing:

1. The separation of discussions on test power under optimal kernel parameters (TST) and test power when the kernel parameters are not optimal (DCT).
2. The demonstration of the effectiveness of the proposed method in the DCT setting.

Additionally, regarding points 1 and 2, I believe that the following conjunction is highly persuasive:

One conjunction. Now, based on Theorem 11, we might have an interesting conjunction. We can assume a scenario where we can obtain the best kernel for MMD DCT (instead of MMD TST) and the best kernel for NAMMD DCT. Based on Theorem 11, if we use the kernel (MMD's best kernel) for NAMMD, then NAMMD DCT will perform better than MMD DCT already. Because is the kernel to make NAMMD DCT have the highest test power (in DCT, instead of TST), NAMMD DCT with should have a higher or equal test power compared to NAMMD DCT with. Thus, NAMMD DCT with has a higher test power than NAMMD DCT with (because NAMMD DCT with has a higher power than MMD DCT with ). We are happy to discuss more with you regading the above content.

Thank you for considering such effective revisions to the research outcomes in such a short period, despite the delay in my response. Thank you again for the highly engaging discussions regarding your research.

Dear Reviewer TXYq,

Many thanks for taking your time to review our responses and providing many insightful comments, which will help us refine and strengthen our work. We acknowledge that our current explanation of the motivation and contributions may not have been as clear as intended, and we sincerely apologize for any confusion/overclaims this may have caused. We would like to provide a clearer and more concise explanation as follows.

To avoid potential overclaims, we will adopt a more cautious tone in the paper. We will emphasize that this paper's analysis applies to kernels with the form $\kappa(x,x')=\Psi(x-x')\leq K$ for a positive-definite $\Psi(\cdot)$ and $\Psi(0)=K$, which includes the Gaussian kernel, Laplace kernel, Mahalanobis kernel, and Deep kernel (some frequently used kernels in kernel-based hypothesis testing). Additionally, we will include a limitation statement regarding the applicability of our findings to other kernels.

We would like first to highlight the following points, which should be very useful to mitigate your concerns. It would be great if you can read them first ^^.

• Existing kernel selection methods are primarily designed for two-sample testing (TST), focusing on selecting the optimal kernel that maximizes the test power estimator of TST to distinguish two fixed distributions $P$ and $Q$ [Sutherland et al., 2017; Liu et al., 2020]. For TST, NAMMD and MMD actually share the same test power estimator because, asymptotically, after we fixed two distributions $P$ and $Q$, NAMMD can be viewed as MMD scaled by a constant $4K-||\mu_P||-||\mu_Q||$. Hence, the NAMMD and MMD has the same optimal kernel for TST. For the same kernel, when we use permutation test to do perform two-sample tests, NAMMD achieves higher test power than MMD due to its scaling as stated in Theorem 9.

• In kernel-based distribution closeness testing (DCT), the distance between a distribution pair ($P_2$ and $Q_2$) is compared to a predefined threshold $\epsilon$, which is determined as the distance between a reference pair of distributions ($P_1$ and $Q_1$). In this case, we use a kernel to calculate the $\epsilon$, and we also need to use the same fixed kernel to calculate the distance for distribution pair ($P_2$ and $Q_2$) to ensure the same distance measurement. However, how to obtain the test power estimator of DCT with multiple distribution pairs and select a global optimal kernel for DCT based on the estimator remains an open question and an important future research.

In this work, we select the kernel following the TST method with the fixed reference distribution pair ($P_1$ and $Q_1$) (on lines 175-179 and Appendix C.1). However, the TST-optimal kernel may exhibit the issue we identified in DCT with MMD, as the kernel selection does not account for distribution pair ($P_2$ and $Q_2$). We propose NAMMD to mitigate this issue by incorporating norm information, $4K-||\mu_P||-||\mu_Q||$, for kernels with the form $\kappa(x,x')=\Psi(x-x')\leq K$ for a positive-definite $\Psi(\cdot)$ and $\Psi(0)=K$, which includes the Gaussian kernel, Laplace kernel, Mahalanobis kernel, and Deep kernel. Since the global optimal kernel cannot be obtained analytically, we prove that, as long as MMD and NAMMD share the same kernel, our NAMMD performs better than MMD as shown in Theorem 11.

One conjunction. Now, based on Theorem 11, we might have an interesting conjunction. We can assume a scenario where we can obtain the best kernel $k^{\rm MD}$ for MMD DCT (instead of MMD TST) and the best kernel $k^{\rm ND}$ for NAMMD DCT. Based on Theorem 11, if we use the kernel $k^{\rm MD}$ (MMD's best kernel) for NAMMD, then NAMMD DCT will perform better than MMD DCT already. Because $k^{\rm ND}$ is the kernel to make NAMMD DCT have the highest test power (in DCT, instead of TST), NAMMD DCT with $k^{\rm ND}$ should have a higher or equal test power compared to NAMMD DCT with $k^{\rm MD}$. Thus, NAMMD DCT with $k^{\rm ND}$ has a higher test power than NAMMD DCT with $k^{\rm MD}$ (because NAMMD DCT with $k^{\rm MD}$ has a higher power than MMD DCT with $k^{\rm MD}$). We are happy to discuss more with you regading the above content.

[1] D. J. Sutherland, H.-Y. Tung, H. Strathmann, S. De, A. Ramdas, A.J. Smola, and A. Gretton. Generative models and model criticism via optimized maximum mean discrepancy. 2017.

[2] F. Liu, W.-K. Xu, J. Lu, G.-Q. Zhang, A. Gretton, and D. J. Sutherland. Learning deep kernels for non-parametric two-sample tests. 2020.

Dear Reviewer TXYq,

Thank you for reading our paper and for your further reviews. Unfortunately, the issue was raised 20 days after the rebuttal period, and we cannot reply through the OpenReview system after the author rebuttal deadline. Although we have reached out to ICLR program chairs via email, we were still unable to resolve this matter.

We address your concern below and hope that future readers will find our clarification helpful. Although the paper has already been rejected, we appreciate the opportunity to provide further insights and contribute to the ongoing discussion with you.

[Q1] In particular, the following test statistic seems to have been used in the calculation of the test power:

According to lines 1172–1174, this test statistic (Equation (E1)) assumes that $\mu_P=\mu_Q=(\mu_P+\mu_Q)/2$, as discussed in lines 848–849. Thus, it appears to be derived under the null hypothesis. However, since the test power is defined as the probability of rejecting the null hypothesis under the alternative hypothesis, the appropriate test statistic should instead be:

[A1] We believe there is a key misunderstanding about the testing thresholds $r_M$ and $r_N$. These thresholds should be determined based on the null distribution [1-4] (the distribution of the test statistics under the null hypothesis, i.e., $H_0: P = Q$ for Theorem 10 ). In contrast,, in your additional comments, we found that you might think the testing thresholds should be from the alternative distribution (the distribution of test statistics under the alternative hypothesis, i.e., $H_1:P\neq Q$), which is incorrect.

To clarify this point, we first introduce the background of hypothesis testing. Hypothesis testing evaluates whether there is enough evidence to reject a null hypothesis $H_0$ in favor of an alternative hypothesis $H_1$. The process involves defining a test statistic $T$ and comparing it to a testing threshold $r$ to determine whether to reject the null hypothesis, where

1. Test Statistic $T$: The test statistic $T$ is a numerical value calculated from the sample data that reflects how much the data deviates from what is expected under the null hypothesis.
2. Testing Threshold $r$: The testing threshold is determined based on the significance level $\alpha$, which represents the probability of rejecting $H_0$ when it is true. Hence, this threshold is often derived from the $(1-\alpha$)-quantile of the (null) distribution of the test statistic under null hypothesis $H_0$.

Hence, the test power (i.e., the probability of correctly rejecting the null hypothesis when the alternative hypothesis is true) is given by

which represents the probability that the test statistic $T$ (calculated under alternative hypothesis where two samples are from different distributions) exceeds the testing threshold $r$ (calculated under the null hypothesis).

Then, we will show why our $r_N$ is correct and why NAMMD has higher test power than MMD. In our paper, for Theorem 10, which addresses the two-sample testing with hypotheses

For MMD test, we have the test statistic $T=m\widehat{\text{MMD}}$ and the asymptotic testing threshold $r_M$, which is the $(1-\alpha)$-quantile of the null distirbution of the test statistic, which is calculated under the null hypothesis $H_0: P=Q$. The corresponding test power is given by

In a similar manner, for NAMMD test, we have the test statistic $T=m\widehat{\text{NAMMD}}$ and the asymptotic testing threshold $r_N$, which is the $(1-\alpha)$-quantile of the null distirbution of the test statistic, calculated under the null hypothesis $H_0: P=Q$. Hence, we have that

meaning that our calculated $r_N$ is correct, and the $r_N$ in your comment is incorrect. Furthermore, the test power of NAMMD test is given by

Consequently, the difference between the test powers of MMD and NAMMD is

This aligns the critical result of our proof in line 1236 and confirms the correctness of Theorem 10.

Building on this, we prove in Theorem 10 that under the alternative hypothesis, if MMD test rejects null hypothesis correctly (i.e., $m\widehat{MMD}$>$r_M$, where $m\widehat{MMD}$ is calculated under alternative hypothesis and $r_M$ is calculated under null hypothesis), our NAMMD test also do so with high probability (i.e., $m\widehat{NAMMD}$>$r_N$, where $m\widehat{NAMMD}$ is calculated under alternative hypothesis and $r_N$ is calculated under null hypothesis). Moreover, NAMMD can reject null hypothesis in cases where MMD fails, resulting in higher test power. This is consistent with our experimental results, where NAMMD demonstrates better performance.

All the below references mentioned that the testing threshold is calculated under the null hypothesis.

[1] W. Jitkrittum, Z. Szabó, K. P. Chwialkowski, and A. Gretton. Interpretable distribution features with maximum testing power. In Advances in Neural Information Processing Systems 29 pages 181–189. Curran Associates, Dutchess, NY, 2016. "The threshold is given by the $(1 − \alpha)$-quantile of the (asymptotic) distribution of $\lambda^n$ under $H_0$, the null distribution."

[2] K. Chwialkowski, A. Ramdas, D. Sejdinovic, and A. Gretton. Fast two-sample testing with analytic representations of probability measures. In Advances in Neural Information Processing Systems 28, pages 1981–1989. Curran Associates, Dutchess, NY, 2015. "the test threshold $T_\alpha$ is given by the $(1 − \alpha)$-quantile of the asymptotic null distribution $\chi^2(J)$"

[3] Gretton, K. M. Borgwardt, M. J. Rasch, B. Schölkopf, and A. Smola. A kernel two-sample test. Journal of Machine Learning Research, 13(1):723–773, 2012. "define a test threshold under the null hypothesis $P = Q$"

[4] F. Biggs, A. Schrab, and A. Gretton. MMD-Fuse: Learning and combining kernels for two-sample testing without data splitting. In Advances in Neural Information Processing Systems36. Curran Associates, Dutchess, NY, 2023. "We can use permutations to construct an approximate cumulative distribution function (CDF) of our test statistic under the null, and choose an appropriate quantile of this CDF as our test threshold"

Looking forward to your further reply!

Best regards,

Feng

Dear Dr. Liu,

Thank you for reaching out and for providing a detailed clarification regarding my concerns. I sincerely regret that my identification of this issue came after the rebuttal period, ultimately leading to a decision where you were unable to fully respond. I understand how frustrating this must be, and I truly appreciate the effort you have put into explaining your perspective.

Because I highly value the work you have put into this research and the effort you have made to clarify your position, I want to ensure that I carefully review your explanation, as well as the references you have cited, to fully understand the theoretical framework you have presented.

While the review process has concluded and the decision was made through discussions among multiple reviewers and the meta-reviewer, I would like to personally provide a well-considered response that not only addresses the specific issue at hand but also offers feedback that may be useful for the future development of your work.

However, due to my current time constraints, I kindly ask for your patience as I take the necessary time to thoroughly re-examine your arguments.

I apologize for keeping you waiting for my response, and I sincerely appreciate your kind understanding.

Best regards,

Reviewer TXYq

Dear Dr. Liu,

I sincerely apologize for the delay in my response. I have carefully reviewed your explanation and the referenced materials you kindly provided, and I have reassessed my original concern regarding Theorem 10 in your paper.

In summary, I remain unconvinced by your explanation. I still believe that the proof of Theorem 10 does not adequately demonstrate the effectiveness of the proposed method. Below, I clarify my position, outline my main arguments, and provide supplementary explanations and suggestions regarding your proof.

1. Clarifying the Points of Agreement

Let me restate my position to avoid any confusion. I fully agree with the following two points you made:

1. Test Statistic $T$:
   The test statistic $T$ is a numerical value calculated from the sample data that reflects how much the data deviate from what is expected under the null hypothesis $H_0$.

2. Testing Threshold $r$:
   The testing threshold is determined based on the significance level $\alpha$, which represents the probability of rejecting $H_0$ when it is in fact true. This threshold is typically derived from the $(1-\alpha)$-quantile of the null distribution of the test statistic.

Given these two points, my concern focuses on the following aspect:

3. Power of the Test:
   The power of the test is the probability of correctly rejecting the null hypothesis, evaluated under the alternative hypothesis $H_1$. In other words, it is the probability that the test statistic $T$ exceeds the threshold $r$ (calculated under $H_0$) when the true distribution is given by $H_1$.

To make my notation clear, let

• $r_M$ be the testing threshold for the MMD-based test (determined under $H_0$),
• $r_N^1$ be $$r_N^1 = \frac{r_M}{4K - \| (\mu_P + \mu_Q)/\sqrt{2} \|^2_{\mathcal{H}\_{\kappa}}}, \tag{E1}$$
• $r_N^2$ be $$r_N^2 = \frac{r_M}{4K - \|\mu_P\|^2_{\mathcal{H}\_{\kappa}} - \|\mu_Q\|^2_{\mathcal{H}\_{\kappa}}}, \tag{E2}$$

where $r_N^1$ and $r_N^2$ are two candidate expressions related to NNMMD.

2. Main Arguments

The fundamental issue with the proof of Theorem 10, as I see it, can be understood via two observations (labeled as R1 and R2 below). These observations suggest that the proposed argument in your paper does not sufficiently establish the advantage of the new method:

• R1. Under the null hypothesis, $r_N^1$ and $r_N^2$ coincide.
• R2. In general, changing only the scale of a test statistic does not alter its power.

I view $r_N^1$ and $r_N^2$ not as the definition of the NNMMD test statistic itself, but rather as (asymptotic) relationships capturing possible scalings between the MMD statistic and the NNMMD statistic. In lines 221–237 of your paper, one can find a more direct definition of the NNMMD test statistic, which I believe should serve as the starting point.

(Addendum on February 24, 2025) Based on point 3 in '1. Clarifying the Points of Agreement', note that Equation (E1) holds only under the null hypothesis and thus cannot be employed when assessing test power under the alternative hypothesis. A relation like Equation (E2), valid under the alternative hypothesis, must be used.

R1. Explanation

Under the null hypothesis $H_0$, $P = Q$. Therefore, the following equality holds:

which implies

Hence, if $r_N^1$ is defined based on the null hypothesis distribution (as you suggest), then $r_N^2$ is equally definable under the same assumption. They coincide under $H_0$.

R2. Explanation

Let me restate the second key idea using an abstract setting. Suppose we have a test statistic $R$ with null distribution $P_{\mathrm{null}}(R)$ and alternative distribution $P_{\mathrm{alt}}(R)$. The original test rejects $H_0$ when

where $\alpha$ is the significance level. Now consider a scaled version of the same statistic, $R / \sigma$. To preserve the same Type I error $\alpha$, the threshold $r'$ must satisfy

Because $\frac{R}{\sigma} > r'$ is equivalent to $R > \sigma \cdot r'$, if we set $r' = r/\sigma$, both tests have the same $\alpha$. Under $H_1$, we then get

Hence, simply scaling the test statistic does not change the power of the test.

References

[1] Gretton, A., Borgwardt, K. M., Rasch, M. J., Schölkopf, B., & Smola, A. (2012). A kernel two-sample test. The Journal of Machine Learning Research, 13(1), 723-773.

[2] Liu, F., Xu, W., Lu, J., Zhang, G., Gretton, A., & Sutherland, D. J. (2020, November). Learning deep kernels for non-parametric two-sample tests. In International conference on machine learning (pp. 6316-6326). PMLR.

[3] Korolyuk, V. S., & Borovskich, Y. V. (2013). Theory of U-statistics (Vol. 273). Springer Science & Business Media.

[4] Lee, A. J. (2019). U-statistics: Theory and Practice. Routledge.

[5] Van der Vaart, A. W. (2000). Asymptotic statistics (Vol. 3). Cambridge university press.

[6] Serfling, R. J. (2009). Approximation theorems of mathematical statistics. John Wiley & Sons.

[7] Hoeffding, W. (1992). A class of statistics with asymptotically normal distribution. Breakthroughs in statistics: Foundations and basic theory, 308-334.

[8] Lehmann, E. L. (1951). Consistency and unbiasedness of certain nonparametric tests. The annals of mathematical statistics, 165-179.

[9] Shekhar, S., & Ramdas, A. (2023). Nonparametric two-sample testing by betting. IEEE Transactions on Information Theory, 70(2), 1178-1203.

[10] Sutherland, D. J., Tung, H. Y., Strathmann, H., De, S., Ramdas, A., Smola, A., & Gretton, A. Generative Models and Model Criticism via Optimized Maximum Mean Discrepancy. In International Conference on Learning Representations.

[11] Gao, R., Liu, F., Zhang, J., Han, B., Liu, T., Niu, G., & Sugiyama, M. (2021, July). Maximum mean discrepancy test is aware of adversarial attacks. In International Conference on Machine Learning (pp. 3564-3575). PMLR.

Best regards,

Feng

Dear Reviewer TXYq,

We are pleased to hear that your concerns have finally been clarified! ^^

To avoid misunderstanding among future readers, may we have your responses to the first public comment (top of comments and just below the abstract) to say your concerns are addressed after the post-decision discussion?

Then, we can safely arxiv this paper without flaw conflicts in public venues.

Best regards,

Feng

Similarly, we test if the ImageNet-V2 is farther from original ImageNet than ImageNet-R. Following Definition 10, we set the original ImageNet as $P_1=P_2$, and set the ImageNet-R as $Q_1$. We further set ImageNet-V2 as $Q_2$ and test if the distance between $P_2$ and $Q_2$ is larger than that between $P_1$ and $Q_1$. The test powers of our NAMMD and MMD with different sample sizes are presented in the table below.