Revisit Non-parametric Two-sample Testing as a Semi-supervised Learning Problem

Thanks for reading our paper and for your valuable feedbacks. We will answer your questions and concerns below.

Weakness 1:

My concern of the learning setting that is proposed in this paper is its difference and benefits compared to positive-unlabeled learning [1] and semi-supervised out-of-distribution detection [2]. Could the authors clarify or include some empirical justifications on the benefits of the proposed learning algorithm?

Response 1:

We appreciate the reviewer’s question about the differences and benefits of the proposed learning setting compared to positive-unlabeled (PU) learning and semi-supervised out-of-distribution (OOD) detection. While we frame two-sample testing as a semi-supervised learning (SSL) problem, we strictly follow the problem setting of two-sample testing, where the goal is to determine whether two given samples are drawn from the same distribution.

This setting makes it challenging to apply PU learning, where the focus is on having labeled data for one class and a large amount of unlabeled data. Similarly, it is hard to align with OOD detection, which focuses on monitoring whether individual data points during operation deviate from the training distribution, without requiring a second group of given samples for comparison. We will further distinguish our setting from both papers mentioned in your comments and cite both papers in discussion, which can help readers understand our paper clearly.

Regarding the effectiveness of the proposed learning algorithm, we would like to address a common misunderstanding about our work. Our aim is to frame two-sample testing as an SSL problem to alleviate the trade-off inherent in the supervised paradigm, where information from unlabeled testing data cannot be fully utilized while maintaining discriminative power.

With this motivation, we initially tested whether existing SSL techniques could be directly applied to two-sample testing but found they failed due to violated assumptions. This led us to explore why these techniques fail and propose a viable method that works specifically in the two-sample testing scenario. We found that those techiques fail because of violation of strict SSL assumptions both empirically and theoretically. Thus, we aim to provide a viable method that our data theoretically holds its assumption. Moreover, the empirical effectiveness of our proposed method—outperforming other two-sample testing methods—has been clearly demonstrated in the paper.

Ultimately, our goal is to pave the way for future research by highlighting the research gap between two-sample testing and SSL, enabling the development of more advanced and adapted SSL techniques for this field. We hope this explanation clarifies the main contribution and the problem setting of our work.

Weakness 2:

What are the key assumptions of the theoretical analysis made in the paper? Are there any empirical justification on their validity on large datasets?

Response 2:

As stated in the paper, our proposed pipeline adds an AE-based representation learning step to the original supervised paradigm. This approach assumes that the data satisfies the manifold assumption that is high dimensional data lies approximately on a lower-dimensional manifold, which is true for all kinds of two-sample testing data.

Regarding empirical justification of the validity on large datasets, none of the ones we provide in our experiments are toy datasets. The HDGM-Hard with dimension 10 is already a challenging large dataset. Even our proposed outperforming method requires 10,000 input samples to reach a test power of 1, demonstrating its capability on large datasets. For the real-life image dataset (MNIST or Imagenet versus their FAKE generated images), both of the distributions contain over 50,000 images. However, since real-life data is generally easier to distinguish than statistically generated edge cases, our method converges to a test power of 1 with fewer than 1,000 drawn samples.

These results validate the assumptions and effectiveness of our method across both large synthetic and real-world large datasets.

We would be delighted to engage in further discussions to ensure your concerns are fully addressed.

Thanks for reading our paper and for your valuable feedbacks. We will answer your questions and concerns below.

Weakness 1:

The datasets used in experiments were from the computer vision domain only. It is not clear that the proposed method will be effective in other domains. At least, the datasets used in the existing study need to be used to demonstrate the usefulness of the proposed method.

Response 1:

We would like to address the concern regarding the datasets used in our experiments. Firstly, our main experiments were conducted on the HDGM datasets, which are widely recognized as a primary benchmark in the two-sample testing scenario. The image datasets, such as MNIST vs. FAKE-MNIST and Imagenet vs. FAKE-Imagenet, were included to demonstrate the practical applicability of two-sample testing methods in real-world scenarios. This combination of synthetic benchmarks and real-life applications ensures a comprehensive evaluation of the empirical power of our proposed approach.

Furthermore, we would like to clarify a potential misunderstanding. The primary focus of this paper is not solely on proposing a new two-sample testing method. Instead, we aim to pave the way for a novel research direction by addressing two key research questions:

1. By intuition, can we directly frame two-sample testing as a semi-supervised learning (SSL) problem?
2. If not, what are the challenges, and how can we propose a viable approach for further research?

Our contribution lies in being the first to propose this perspective and providing insights into how SSL can address the trade-offs and limitations in existing two-sample testing methods.

We hope this clarification addresses the concern and highlights the significance of our research questions in advancing the field.

Question 7:

From the SSL setting viewpoint, the proposed method can be compared with SSL methods using, e.g., 1,000 labeled and 10,000 unlabeled samples.

Response 7:

We appreciate the reviewer’s suggestion regarding comparisons under the SSL setting with configurations such as few labeled samples and lots of unlabeled samples. However, we would like to clarify that the two-sample testing scenario differs fundamentally from the traditional SSL setting.

In two-sample testing, we can assume that all input data is labeled because the goal is to determine whether two given samples, $S_P$ and $S_Q$, are from the same distribution. The challenge lies in the trade-off of splitting proportions between training and testing sets, as also required by the supervised paradigm. Moreover, this splitting proportion will not influence the representation learning (we use all the input data for representation learning).

While the trade-off of splitting proportions is a central issue in two-sample testing, it is not the primary focus of our work. Instead, we aim to explore whether and how the information from the testing set (treated as unlabeled data) can be utilized to enhance test performance. This conceptual shift is what we aim to demonstrate as viable and valuable in this work. We hope this explanation clarifies why traditional SSL data configurations are not directly applicable in the context of two-sample testing and how our work approaches this unique challenge.

Question 8:

In Sec 5.2, the setting in this paper implies m_l=m_u and labeled samples are identical to unlabeled samples. Applying the existing analysis to SSL may not be valid since the assumptions of the existing analysis and this paper are probably different. This paper should carefully discuss such a difference in the assumption.

Response 8:

We would like to clarify that $m_l$ and $m_u$ are not identical in our setting. The unlabeled sample size $m_u$ must be greater than $m_l$, as it includes all the training data (with labels removed) and all of the testing data.

Given this, we believe there is no additional assumption required to discuss, as our setting aligns with the framework used in the analysis.

We would be delighted to engage in further discussions to ensure your concerns are fully addressed.

Dear Reviewer 4szV,

Thanks for proposing your another concern. Since $\psi$ is irrelevant to the f, we think there is no need to include $\psi$ into the compatibility notation. Just same as other semi-supervised learning techniques, like consistency regularisation, it will only show that the unsupervised loss to be $\frac{1}{N} \sum_{i=1}^N \|f(x_i) - f(a(x_i))\|^2$, since the augmentation function $a$ is irrelevant to the f, so there is no need to specify how $a$ related to the compatibility. Hope this addresses your concerns, if all your concerns have been resolved, we kindly ask if you could consider revisiting your scores and updating them to reflect the clarified contributions of our work.

Best regards,

Authors of Submission 6635

Thanks for reading our paper and for your valuable feedbacks. We will answer your questions and concerns below.

Weakness 1:

The overall presentation and organization of the paper is very poor. The actual details of the method appears quite late on page 6. The introduction section is unreasonably long.

Response 1:

We appreciate your feedback regarding the presentation, organization, and the length of the introduction. However, we believe there is a significant misunderstanding about the primary focus and contribution of our work, which we would like to clarify here:

Our main focus / contribution is not proposing a single new method. Instead, we are the first to introduce the perspective that the two-sample testing problem (which is a specific case within the field of hypothesis testing) can be framed as a semi-supervised learning (SSL) paradigm. This novel conceptual contribution addresses the limitations of data-splitting trade-offs inherent in supervised methods and the lack of discriminative power in purely unsupervised methods. This is clearly stated in the 'Two learning paradigms and their issues' section of the Introduction on Page 2 and elaborated further in Section 3 on Page 5.

After developing the intuition that we can combine the field of two-sample testing with the field of semi-supervised learning, since we are the first to propose this novel view, to validate whether this combination is viable, we investigate whether existing SSL methods can directly apply to two-sample testing in the Section 4 on Page 5-6. After investigation and experimental verification, existing SSL methods fail due to specific challenges posed by overlapping data distributions in two-sample testing. Identifying and explaining this limitation is also a critical part of our contribution.

Based on the findings of our survey and analysis, we propose a viable way to adapt SSL for two-sample testing, which serves as a first step for future research. While this method is simple yet effective, the conceptual contribution lies in paving the way for future advancements by adapting more advanced SSL methods to the two-sample testing field.

Our contribution is to establish a new research direction that bridges the gap between two previously separate fields, and point out that main limitation of the current supervised paradigm which will discard the information from the unlabeled testing data. We hope this clarification helps to better understand our work and its importance in advancing the field of two-sample testing and hypothesis testing.

Weakness 2:

Figure 2 looks unnecessary. Figure 1 and Figure 3 are repeated.

Response 2:

We would like to kindly point out that the importance and necessity of each figure are already explained in the paper. We encourage the reviewer to carefully revisit the respective sections where these figures are discussed. Below, we further clarify their distinct roles to avoid any misunderstanding.

Figure 2 is critical as it visualizes synthetic data scenarios to mimic the types of data distributions we encounter in two-sample testing experiments. It highlights the degree of data overlap, which directly impacts the viability of state-of-the-art (SOTA) SSL methods for two-sample testing. By clearly showing how the distributions differ and overlap, Figure 2 provides essential context for understanding the challenges our method addresses.

Figure 1 provides a high-level overview of the existing paradigms in two-sample testing: unsupervised, supervised paradigm. This figure helps readers conceptually understand the differences between the approaches, framing the motivation for introducing SSL in the context of two-sample testing.

Figure 3 focuses specifically on our proposed SSL-C2ST pipeline, showing how it integrates the strengths of the original C2ST paradigm while leveraging the information from the unlabeled testing data. Unlike Figure 1, which provides a broader summary, Figure 3 explains the detailed implementation and workflow of our proposed method.

Each figure has a unique purpose and contributes to the overall understanding of our work. We believe their inclusion is essential for a clear and comprehensive presentation of our methodology and experimental setup. We hope this explanation clarifies the necessity and distinct roles of these figures.

Question 1:

Definition of χ in Definition 5.2 is not clear. How is χ(f,x) defined ? Proof of Theorem 5.3 is also very unclear. What is C[2m1,S], the notation is not defined. m1 is also not defined (although it could be inferred that m1 is number of labeled data).

Response 1:

Firstly, χ represents the compatibility, which is clearly defined in Definition 5.2 on Page 8 and directly refers to the work by [1] under our transductive learning settings. We encourage the reviewer to revisit this definition for further clarification.

The notation C[2$m_l$, S] is defined around line 430 as the expected split number for 2$m_l$ points drawn from S using functions in C. Additionally, it is m_l (the labeled sample size), not m_1 as inferred. This is explicitly defined on line 428 of the paper.

We hope this clarification resolves the concerns regarding the definitions and proof notations. Please let us know if further elaboration is required.

[1] Maria-Florina Balcan and Avrim Blum. A discriminative model for semi-supervised learning. Journal of the ACM, 57(3):19:1–19:46, 2010

Question 2:

Not clear from the theorem how this gives more power than C2ST ?

Response 2:

Please refer to the Response 6 to Weakness 6.

Question 3:

“Although the differences between two methods are little when N is small, the test power of SSL-C2ST has a huge gap over C2ST when N is large enough” This only seems to be the case for the right most figure ?

Response 3:

As you can see clearly from the y-axis of Figure 4, the gap appears larger in the right-most subfigure because HDGM-Hard with dimension 10 is a more challenging dataset, making it harder for C2ST to converge to 1 with a relatively small number of input samples compared to SSL-C2ST.

For the other two relatively easier datasets shown in the other subfigures, the huge gap lies in the number of samples required for convergence. While SSL-C2ST converges to 1 with relatively fewer input samples, C2ST requires a significantly larger amount of data to achieve the same convergence. This difference reflects the performance gap between the two methods.

In two-sample testing, under the alternative hypothesis, as the number of samples increases, the test power of an advanced model will eventually converge to 1. If a model requires significantly fewer samples to reach this convergence, it is considered to be outperforming.

We hope this explanation provides additional clarity regarding the observed gaps in Figure 4.

We would be delighted to engage in further discussions to ensure your concerns are fully addressed.

Dear Reviewer AuP3,

Thanks for your reply. For compatibility, it is indeed defined in Definition 5.2 on Page 8. We are not sure if you indicated to give some examples regarding this function. If so, we are happy to give it to you. it is just a function, and we defined its input and output spaces already.

As for the novelty, we respectfully disagree with you, and your judgment is subjective. Our main contribution lies in that we find a new perspective to study the two-sample test, where the two-sample test and corresponding techniques are one of the most important tools used in the machine learning field. Thus, proposing a new research direction for the two-sample test, an important problem, is novel and significant. This will light up more research studies in this field, by following our novel perspective.

Our novelty is fully recognized by three reviewers:

1. Reviewer Y9aR: "The paper is novel in applying a semi-supervised learning framework to the two-sample testing problem, a first in this area. The proposed framework is simple yet effective, as demonstrated by the experimental results."

2. Reviewer 6c7X: "The authors clearly highlight the issues with the current methodologies, particularly when viewed from the bucket of supervised and unsupervised methods. The proposed approach cleverly circumvents this problem for both buckets. The applicability to different scenarios seems to be the biggest plus point of using SSL based testing methods."

3. Reviewer 47NZ: "This paper proposes a novel perspective to consider non-parametric two-sample testing as a semi-supervised learning (SSL) problem."

We hope to further discuss with you and other reviewers the position of our paper in the field of two-sample testing. We believe this is the major divergence between you and us. Thus, it is important to find a consensus toward the novelty judgment.

Best regards,

Senior Author of Submission 6635

Thanks for reading our paper and for your valuable feedbacks. We will answer your questions and concerns below.

Weakness 1:

Writing: The limitations of DRs and IRs, along with the motivation for the proposed method, are reiterated throughout Sections 1 and 3 without providing additional insights. Although recognizing these problems is key to the paper’s contributions, a single, well-placed explanation could make the writing more concise.

Response 1:

We appreciate the reviewer’s feedback regarding the writing and organization of Sections 1 and 3. We agree that emphasizing the limitations of supervised and unsupervised paradigms is key to motivating our work, but we see an opportunity to improve conciseness.

To address this, we will reorganize Section 3 to provide a concise summary of these limitations and motivations, followed by the pointing out the two research questions we aim to address. This adjustment will make the paper more organized and easier to follow.

Weakness 2:

Novelty: The proposed SSL-C2ST pipeline is straightforward and highly effective, but it closely resembles standard fine-tuning practices, where only the last few layers are updated after learning self-supervised features — a common approach in neural network training.

Response 2:

We appreciate the reviewer’s observation regarding the simplicity of the SSL-C2ST pipeline. As we restate in the paper, the main goal of our work is to pave the way for future research on framing semi-supervised learning techniques within the two-sample testing field, e.g. testing the viability of directly applying SOTA SSL techniques on the two-sample testing methods, proposing a simple but viable method.

Our primary contribution is to demonstrate that this perspective is worthy of further exploration and that even a simple method can effectively improve performance in this context. We also highlight that methods with strict assumptions often struggle in two-sample testing scenarios, making this an important area for developing more advanced, adapted SSL techniques.

Weakness 3:

Ambiguity: It is not very clear if obtaining features in an unsupervised manner is classified as SSL. Related to this, what do the authors think the underlying assumption of auto-encoder features is for SSL tasks if none of the three assumptions apply for this?

Response 3:

We would like to clarify the assumptions regarding two-sample testing data and their relevance to our method. As discussed in the paragraph "Testing data might not satisfy the assumptions made by many SSL methods" on Page 6, the data violates the smoothness and cluster assumptions but satisfies the manifold assumption.

The choice of autoencoder-based representation learning is motivated by two main considerations:

1. Alignment with C2ST architecture: In the original C2ST model, a feature extractor is included. By incorporating an autoencoder, we leverage the additional unlabeled testing data to enhance the feature extractor’s capability.

2. Relaxed assumptions: Unlike many other SSL techniques, which depend on fulfilling multiple assumptions (e.g., smoothness, cluster, and manifold), the effectiveness of an autoencoder primarily relies on the manifold assumption, which is satisfied in two-sample testing scenarios.

Additionally, we would like to restate the original definition of SSL setting, as defined in [1]: SSL can leverage the information P(x) from unlabeled data to help the inference of P(y|x). If the unlabeled data degrades prediction accuracy by misguiding the inference (e.g., due to violating the assumptions of SSL techniques), then that cannot be classified as effective SSL method. This definition is critical because it is aligned with our SSL setting, which we utilize the information of unlabeled data to improve the prediction accuracy.

We acknowledge that the original definition of SSL might not have been clearly reiterated in the paper. We will ensure to emphasize it and restate the reasons for using AE-based representation learning in the revised version.

[1] Olivier Chapelle, Bernhard Schölkopf, and Alexander Zien (eds.). Semi-Supervised Learning. The MIT Press, 2006. ISBN 9780262033589.

Question 1:

Could you elaborate the equivalence between $\max_{\epsilon} (\frac{1/2 - \epsilon}{\sqrt{\epsilon - \epsilon ^2}})$ and $\min_{\epsilon} (\frac{\epsilon}{\sqrt{\epsilon - \epsilon ^2}})$? It is not very obvious to me because of the denominator.

Response 1:

Sorry for a direct transformation of the equation, and we will provide the detailed explanation of the equivalence.

If we define $f(\epsilon) = \frac{1/2 - \epsilon}{\sqrt{\epsilon - \epsilon ^2}}$, $g(\epsilon) = \frac{\epsilon}{\sqrt{\epsilon - \epsilon ^2}}$ and $D(\epsilon) = \sqrt{\epsilon - \epsilon^2}$, where $\epsilon \in (0, \frac{1}{2})$.

The equation $\max_{\epsilon} f(\epsilon) = \max_{\epsilon} \left(\frac{1/2}{D(\epsilon)} - g(\epsilon)\right)$ holds. It is clear to find that $f(\epsilon)$ and $\frac{1/2}{D(\epsilon)}$ are monotonically decreasing over the domain of $\epsilon$. Thus, only if $g(\epsilon)$ is monotonically increasing over the domain of $\epsilon$, the equation $\max_{\epsilon} \left(\frac{1/2}{D(\epsilon)} - g(\epsilon)\right) = \max_{\epsilon} \left(\frac{1/2}{D(\epsilon)}\right) - \min_{\epsilon} \left(g(\epsilon)\right)$ holds. We can understand the ambiguity that why the numerator and denominator of $g(\epsilon)$ are both increasing functions, but we declare the monotonic increasing behavior of entire $g(\epsilon)$. The reason why we say that is over the domain of $\epsilon$, we can find that the increasing rate of numerator is larger than that of denominator. Moreover, we can provide the proof of this phenomenon by simply calculating the derivative of $g(\epsilon)$ w.r.t $\epsilon$. Firstly, let us calculate the derivative of $D(\epsilon) = \left(\epsilon - \epsilon^2\right)^{1/2}$ w.r.t $\epsilon$,

$ D'(\epsilon) &= \frac{1}{2(\epsilon - \epsilon^2)^{1/2}} \cdot (1 - \epsilon + (-\epsilon)) \\\\ &= \frac{1 - 2\epsilon}{2D(\epsilon)}, $

then, we take the derivative of $g(\epsilon) = \frac{\epsilon}{D(\epsilon)}$ w.r.t $\epsilon$,

$ g'(\epsilon) &= \frac{1 \cdot D(\epsilon) - \epsilon \cdot D'(\epsilon)}{D(\epsilon)^2} = \frac{1}{D(\epsilon)^2} \cdot \frac{2D(\epsilon)^2 - \epsilon(1-2\epsilon)}{2D(\epsilon)} \\\\ &= \frac{1}{\epsilon(1-\epsilon)} \cdot \frac{2(\epsilon - \epsilon^2) - \epsilon(1-2\epsilon)}{2D(\epsilon)} \\\\ &= \frac{\epsilon}{\epsilon(1-\epsilon) \cdot 2\sqrt{\epsilon(1-\epsilon)}} = \frac{1}{2(1-\epsilon)\sqrt{\epsilon(1-\epsilon)}}. $

We can find that over the domain of $\epsilon \in (0, \frac{1}{2})$, $g'(\epsilon) > 0$, which concludes the proof. The reason why we want to derive the objective to be equivalent to $\min_{\epsilon}\frac{\epsilon}{\sqrt{\epsilon - \epsilon ^2}}$ is we can simplify it to $\min_{\epsilon}\sqrt{\frac{\epsilon}{(1-\epsilon)}} = \min_{\epsilon}\frac{\epsilon}{(1-\epsilon)}$, where $\epsilon \in (0, \frac{1}{2})$. In that way, it is quite straightforward to understand how minimizing $\epsilon$ can help to improve test power.

In order to eliminate ambiguity, we will also add this proof in the revised paper.

Question 3:

Auto-encoder is not the only way to obtain unsupervised-learning features. In fact, contrastive learning is perhaps more popular and effective. Is there any particular reason why the authors restricted unsupervised-learning features to be auto-encoder-based features?

Response 3:

We appreciate the reviewer’s question about the choice of auto-encoder-based features for unsupervised representation learning. As we mentioned in our response to Weakness 3, auto-encoder-based representation learning was chosen because it is effective when the manifold assumption holds, which is the most suitable for two-sample testing data.

On the other hand, other popular unsupervised representation learning techniques, such as SimCLR [1] and SwAV [2], which still utilize the point-wise semi-supervised learning techniques such as label propagation or sample augmentation. These approaches are not directly viable in the field of two-sample testing due to the violation of the smoothness assumption and cluster assumption in this context. Moreover, if we preserve the sample index of two samples $S_P$ and $S_Q$, then conduct a representation learning to learn a feature extractor that can best separate two samples, we will face a problem in the hypothesis testing is that the type-I error will be out of control, since using unlabeled testing data to learning DRs will always make the probability of successfully discriminating two samples from the same distribution be higher than the significance level $\alpha$.

We hope this clarifies why auto-encoder-based representation learning was selected as the most suitable method for our framework. Moreover, we will add the above discussion into the revised paper to help readers to better understand the motivation why we choose AE-based representation learning rather than unsupervised contrastive representation learning.

[1] Chen T, Kornblith S, Norouzi M, Hinton G. A simple framework for contrastive learning of visual representations. InInternational conference on machine learning 2020 Nov 21 (pp. 1597-1607). PMLR.

[2] Caron M, Misra I, Mairal J, Goyal P, Bojanowski P, Joulin A. Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems. 2020;33:9912-24.

We would be delighted to engage in further discussions to ensure your concerns are fully addressed.

Dear Reviewer Y9aR,

Thank you for your valuable feedback and constructive comments on our submission.

Since we have already solve all the concerns that has been proposed, we would like to ask if you have any additional concerns or questions about the paper that we can address. If there are specific points that remain unclear, please feel free to point them out—we are more than willing to discuss and provide further explanations.

If all your concerns have been addressed satisfactorily, we would sincerely appreciate it if you could consider revisiting your scores and generously increasing them, if appropriate, to reflect the clarified contributions of the paper.

Thank you once again for your thoughtful review and for helping us improve our work.

Best regards,

Authors of Submission 6635

Dear Reviewer Y9aR,

Many thanks for recognizing our contributions to the field. We think you have concluded our contributions clearly:

This work is valuable in informing the community about the effectiveness of autoencoder-based SSL for two-sample testing, while highlighting that a naive application of other SSL methods may not yield similar benefits.

Novelty concern. As for the novelty of our method, we want to emphasise that our main contribution is to investigate how SSL works for two-sample testing problem at its first place. It is always difficult to be the first work to pave a new perspective to understand an important problem, and we did a thorough investigation on this point in the current paper.

Best regards,

Authors of Submission 6635

Question 1:

For Table 1, I understand there is no standard deviation across one sampling of two groups, but would there not be several runs of such samplings that could be aggregated against and then the standard deviation be expressed? While the result of each trial is 0 or 1, would it not be more insightful to have several trials (say 5) and report the results across them in total?

Response 1:

We appreciate the reviewer’s question regarding the reporting of standard deviation in Table 1. To clarify, we indeed conducted each experiment 100 times, or to say sampling 100 trials from the two distributions. However, in each sampling, the result is either 0 or 1, meaning that the standard deviation becomes quite large and less meaningful when the input sample size is small. Moreover, if you suggest to perform the permutation testing several times, we also indeed perform permutation testing 100 times for each sampling, but since our problem setting is transductive learning, so the result is still either 0 or 1, or very close to these two values.

To address this, we only use test power to represent the results. Test power reflects the estimated probability of successfully discriminating (i.e., returning 1) between the two samples across the 100 trials. This provides a clearer and more interpretable metric for our experiments.

We hope this explanation clarifies our choice of representation in Table 1.

We would be delighted to engage in further discussions to ensure your concerns are fully addressed.

Thanks for reading our paper and for your valuable feedbacks. We will answer your questions and concerns below.

Weakness 1:

I think some reorganization might help the paper read better, for example including the Algorithm 1 in the main paper perhaps at the cost of some preliminaries or shortening the introduction could help understand the proposed method better. Right now, while the motivation and setup comes across very clearly, the details of the approach are somehow missed, and the key points of why the method works in practice, while written in words, are not evident algorithm wise.

Response 1:

We appreciate the reviewer’s suggestion regarding the paper’s organization, and we agree that reorganizing Section 3 which has some repeated contents to the introduction could improve the paper's readability. Specifically, we can shorten the main paper by concisely restating the key motivation and emphasizing the two research questions addressed in this paper.

The freed-up space can then be used to include Algorithm 1 in the main paper, which will provide clearer support for the details of the approach and make the key points of why the method works in practice more evident. We believe these adjustments will make the paper more organized and easier to follow while addressing the concern about missing algorithmic details.

Weakness 2:

On a similar note, I’m slightly unsure where the exact improvements are coming from, is it possible to look at some ablations where one of the steps was modified, say the featurizer for learning IRs was an suboptimal one, how would the learning DRs be affected? It would be helpful to understand the utility of each part of the sequential method proposed by the authors.

Response 2:

The improvement in performance comes from the fact that the feature extractor learned from the autoencoder will be effective when our data well fits the manifold assumption, that is high dimensional data lies approximately on a lower-dimensional manifold. Moreover, effective and accurate lower-dimensional data representation will improve the capability of detecting discrepancy in the MMD testing. As found in [1], the faster the actual dimension decreases relative to the sample size, the higher-order moment discrepancy the MMD test are capable to detect, so MMD-based SSL-C2ST (SSL-C2ST-M) can be such outperforming than the other SOTA two-sample testing methods across all the benchmarks.

For the ablation study, as shown in Figure 3, compared to the original C2ST pipeline, our improvement lies in pre-training the featurizer on the entire labeled and unlabeled dataset, which will learn a better IRs than that of C2ST. This step allows the featurizer to become more general and capable of mapping the data onto an effective low-dimensional manifold, which facilitates the following learning discriminative representations (DRs). The utility of this additional step is also demonstrated empirically in the Figure 4 in the experiments section, where we show how incorporating this pre-training step leads to improved test power than that of the original C2ST.

We hope this explanation clarifies the utility of each part of the sequential method and the source of the observed improvements.

[1] Yan J, Zhang X. Kernel two-sample tests in high dimensions: interplay between moment discrepancy and dimension-and-sample orders. Biometrika. 2023 Jun 1;110(2):411-30.