Practical Kernel Learning for Kernel-based Conditional Independent Test

Q4. In addressing the causal discovery problem, you have only presented the F1 scores for the POWER and Median methods, and the results appear to lack significance. Could you conduct additional experiments to provide further clarification?

A. Thank you for your suggestion. We further evaluated our method and the Median approach on two real-world benchmarks (SACHs and CHILD), using the same experimental parameters as those in our synthetic dataset experiments. The results are as follows. From the results, it can be observed that on these two graphs, our method also outperforms the Median-based approach.

Table 1: SACHS Results

Table 2: CHILD Results

Q5. Could you provide a theoretical explanation of how this method enhances statistical power?

A. Thank you for your comments. The theoretical analysis of test power involves the estimation bias of CME in our method, which requires further investigation. We appreciate you providing us with a good alternative approach for analyse, and we will attempt further theoretical analysis in this direction.

Q6. Did the authors evaluate the effects of data splitting on their methodology through simulation experiments to determine whether it is necessary to perform data splitting?

A. Our method indeed requires splitting the samples, as it involves not only estimating the conditional mean but also learning the parameters of the statistic itself (specifically, the kernel parameters applied to the conditioning set). Below is an experiment demonstrating the impact of data splitting on our method. It shows a significant increase in Type I error when the data is not split, indicating that the learned parameters are overfitted to the training sample size. Therefore, splitting the data is essential for the validity of our method. In the experiment, the training set and test set were evenly split.

Table: Comparison of Type I and Type II Errors (Split and Non-Split)

We sincerely appreciate your profound and valuable suggestions for our paper. Thank you for your effort and dedication!

W1. The method presented in this paper demonstrates only marginal improvements in power, yielding minimal gains compared to the original median heuristic algorithm.

A. We believe that evaluating our method requires considering its minimal additional computational overhead. Experimental results show that our method consistently outperforms the Median heuristic-based approach across different dimensions of the conditioning set. This also demonstrates that our method can effectively select more suitable kernel parameters.

W2. & Q1. The conditional mean of the statistic is estimated using kernel ridge regression; however, the paper derives the asymptotic normality of the oracle statistic only under the case when the conditional mean is known. If the estimated conditional mean is incorporated into the statistic, does it still retain asymptotic normality?

A. Our current method uses Gaussian processes to estimate the conditional mean. While it does have estimation bias, the estimation error decreases as the sample size increases (even though its learning rate is complex), which improves the accuracy of the test power. Therefore, in cases with larger sample sizes, we can still utilize the approximate testing power derived from this property for parameter learning. From a theoretical perspective, due to the unclear estimation error of the CME, further research is required. This will be our future research direction. Thank you very much for your suggestion.

W3. & Q3. Your method seems to overlook scenarios involving higher dimensionality of Z and smaller sample sizes. For example, dz=100 and n=500 as in Li et.al. (2024).

A. Thank you for your suggestion. We followed the experimental setting in Li et al. (2024) to test performance under different sample sizes and compared it with the baseline proposed by Li et al. The results are as follows. Except for Li's method, which retains all default settings, the data in rest methods was evenly split into training and test datasets. From the results, it can be observed that, even in high-dimensional scenarios, under certain specific settings, the kernel-based methods still demonstrate comparable Type I error control and test power.

Type I Error with different sample size N ($\alpha = 0.05$)

Type II Error

Q2. Generative models can serve this purpose such as Sinkhorn GAN for estimating conditional means. If these methods were to replace kernel ridge regression for estimating conditional means, what impact would this have on your approach?

A. Thank you for your suggestion. Our method regularizes CME in the RKHS space instead of in the real number space. Using GANs to estimate the theoretically infinite-dimensional CME may require further consideration.

We sincerely appreciate your profound and valuable suggestions for our paper. Thank you for your effort and dedication!

W1. I think the main weakness of this paper lies in its novelty. I don't see any specific methodological designs that incorporate this when optimizing kernel parameters.

A. Thank you for your comment. It is true that using test power to optimize kernel parameters has been widely adopted since Gretton et al. (2012) proposed it. However, to the best of our knowledge, the choice of kernels in CI tasks has not been thoroughly studied. Generally, this methodology optimizes all kernel parameters involved in the statistic. However, due to the unique nature of CI tasks (where the estimated conditional mean embedding inherently contains bias with limited samples), we found that directly optimizing the original KCI statistic does not effectively improve its test power.

Instead, we took a surrogate approach by decomposing the original KCI statistic and directly exposing the kernel parameters applied to the conditional variable for optimization. This partially improved the test power. While we cannot claim to have found the optimal kernel parameters as Gretton et al. (2012) did, this approach avoids the redundancy of re-learning the conditional mean embedding by selecting the kernel parameters on the response variable. In terms of results, our method achieves higher test power with minimal computational overhead and comparable Type I error control, which represents our improvement.

W2. The proposed method does not appear to show significant improvement based on the results presented in Figures 1 and 2.

A. We aim to propose a kernel parameter selection method for KCI that outperforms the median heuristic. As shown in Figure 1, our method consistently performs better than the median heuristic across different dimensions and sample sizes with comparable Type I control. Furthermore, considering the minimal additional computational overhead of our approach, we believe that our method can replace the original median heuristic-based KCI in most cases.

Q1. What is the convergence rate of these estimated parameters? Do they affect the asymptotic variance in equation (6)? Are there any theoretical guarantees of the proposed method?

A. Due to the presence of conditional embedding bias, whose convergence rate remains a challenge, the convergence rate of the estimated statistic and the parameters involved is actually influenced by the convergence rate of the conditional expectation and, therefore, is still under further investigation. However, as shown in the experimental results, under a reasonably large sample size, our method can use the estimated test power based on the asymptotic distribution to select more suitable kernel parameters using the estimated power. We will focus more on the convergence rate of these parameters in future work. Thank you for your comments.

Q2. In Section 2.1, the authors mention the necessity of a balanced assessment of both Type I and Type II error rates in the practical evaluation of CI methods. While I acknowledge the difficulty of CI testing problems, as highlighted by Shah and Peters (2020), I argue that controlling Type I error rates is paramount. Under what conditions can Type I error be effectively controlled by the proposed method?

A. Thank you for your comments. "The necessity of a balanced assessment of both Type I and Type II error rates in the practical evaluation of CI methods" are grounded in a "practical" perspective. On one hand, CI testing inherently differs from other hypothesis testing tasks, such as two-sample tests or independence tests, because it specifically considers whether there is a shared latent confounder among observed variables. This involves testing the independence of latent variables, requiring the exposure of latent variables (which introduces misspecification bias due to limited samples in practice). Thus, without assumptions, no method can effectively control the Type I error while maintain valid test power when dealing with general CI cases in practice. On the other hand, while some approaches like GCM (Shah et al., 2020) and CRT (Candes et al., 2018) claim to effectively or asymptotically control Type I error, they often come with stringent conditions (e.g., assuming the conditional distribution is known or the in-sample mean-squared error is small), which is almost impossible to achieve or verify in practice.

In our method, due to the estimation of CME involved, which affects the control of Type I error, its convergence behavior is still unclear and requires further investigation. But from a practical perspective, our method empirically allows for better selection of kernel parameters across general cases. This demonstrates that even in the presence of CME estimation bias, our method remains effective and achieves better test power compared to the Median approach from a practical perspective.

We sincerely appreciate your profound and valuable suggestions for our paper. Thank you for your effort and dedication!

W. The idea of learning kernel functions or kernel parameters is not new in the literature.

A. Thank you for your comment. Using test power to optimize kernel parameters has been widely adopted since Gretton et al. (2012) proposed it. However, to the best of our knowledge, the choice of kernels in CI tasks has not been thoroughly studied. Generally, this methodology optimizes all kernel parameters involved in the statistic. However, due to the unique nature of CI tasks (where the estimated conditional mean embedding inherently contains bias with limited samples), we found that directly optimizing the original KCI statistic does not effectively improve its test power.

Instead, we took a surrogate approach by decomposing the original KCI statistic and directly exposing the kernel parameters applied to the conditional variable for optimization. This partially improved the test power. While we cannot claim to have found the optimal kernel parameters as Gretton et al. (2012) did, this approach avoids the redundancy of re-learning the conditional mean embedding by selecting the kernel parameters on the response variable. In terms of results, our method achieves higher test power with minimal computational overhead and comparable Type I error control, which represents our improvement.

[1] Gretton A, et al. Optimal kernel choice for large-scale two-sample tests[J]. NeurIPS, 2012.

Q2. It is unclear which parts are original contributions by the authors and which are drawn from existing literature. Please clarify this distinction, and provide appropriate references for any cited results.

A. Thank you for your comments. In future revisions, we will more appropriately cite relevant papers and better highlight our contributions.

Q3. Does the proposed method learn only the kernel bandwidths, or does it also learn the kernel functions?

A. Our method currently focuses on learning the parameters within a given kernel family (Gaussian kernel). Theoretically, our approach could also be extended to select between different kernel families (which might require further discussion on details such as whether the true power remains consistent for the same estimated power across different kernel families). However, as we stated in the paper, "This selection process primarily focuses on tuning kernel parameters, which can often be more influential than the choice of the kernel family itself." Different kernel families can exhibit similar filtering characteristics. Therefore, for now, we are concentrating on the former.

Q4. The asymptotic results are derived under fixed kernel parameters. How do the learned parameters affect these results? Moreover, how can the authors ensure that the proposed method at least asymptotically controls the type I error using the weighted sum of chi-squared to compute the p-value?

A. Our method does not affect its asymptotic properties, regardless of whether the weighted sum of chi-squared or other approximation methods is used. The difference from the median heuristic-based one lies in how to determine kernel parameters involved before testing: the median heuristic uses a heuristic approach, while we propose determining them based on the estimated test power. During the testing phase, all kernel parameters in our method are fixed with the pre-split test samples. Therefore, this does not alter its asymptotic properties.

Q5. Could the authors provide a more detailed explanation of the impact of the conditional expectation bias?

A. The misspecification of the conditional embedding can lead to residuals that still exhibit correlation induced by the conditioning variable. This correlation bias not only causes the shift in the mean of the statistic with limited samples (leading to uncontrollable Type I error) but also affects the estimation of test power. Considering this bias, we opted to fix the residual part. Although the bias remains, it is fixed. As the sample size increases, the estimated test power gradually converges, enabling the use of the estimated power for the selection of more appropriate kernel parameters.

We sincerely appreciate your profound and valuable suggestions for our paper. Thank you for your effort and dedication!

W1. If the statistical method proposed is not able to control type I error, at least under some assumptions, how can it be considered a valid testing procedure?

A. Thank you for your insightful evaluation. The points regarding "no method can simultaneously control the Type I error rate at the given significance level while maintaining adequate power" and "the practical evaluation of CI methods necessitates a balanced assessment of both Type I and Type II error rates, emphasizing the trade-off between error control and statistical power" are grounded in a "practical" perspective.

On one hand, CI testing inherently differs from other hypothesis testing tasks, such as two-sample tests or independence tests, because it specifically considers whether there is a shared latent confounder among observed variables. This involves testing the independence of latent variables, requiring the exposure of latent variables (which introduces misspecification bias due to limited samples in practice). Thus, without assumptions, no method can effectively control the Type I error while maintain valid test power when dealing with general CI cases in practice.

On the other hand, while some approaches like GCM (Shah et al., 2020) and CRT (Candes et al., 2018) claim to effectively or asymptotically control Type I error, they often come with stringent conditions (e.g., assuming the conditional distribution is known or the in-sample mean-squared error is small), which is almost impossible to achieve or verify in practice.

Therefore, we believe that in real-world tasks, an empirical evaluation for general CI testing still necessitates a comprehensive assessment from the perspectives of both "error control and testing power" in broad-ranging cases. This is also why we describe our approach as a "practical kernel choice method."

W2. On the Type I error performance in synthetic data and high-dimensional data.

A. As mentioned in the response to Q1, KCI-based methods, whether CIRCE or our approach, cannot fully control Type I error. Therefore, from a practical perspective, it is still necessary to simultaneously compare both Type I error and Type II error.

In high-dimensional experiments, we followed the experimental setting in Polo et al. (2023) to compare the performance of different methods under such extreme conditions. The results show that, even when all methods perform poorly, our approach still slightly outperforms the Median heuristic.

We also followed another high-dimensional experiment from Li et al. (2024) We also followed another high-dimensional experiment from Li et al. (2024) (following the suggestion from Reviewer QG5Q). In this experiment, the conditioning set dimension was set to 100 , and the results are as follows. And KNN is the proposed method in Li et al. From the results, it can be observed that, even in high-dimensional scenarios, under certain specific settings, the kernel-based methods still demonstrate comparable Type I error control and test power.

Type I Error with different sample size N ($\alpha = 0.05$)

Type II Error

Q1. An operator "is not Hilbert-Schmidt in characteristic RKHS" does it means that is not bounded?

A. Yes, because when $\mathcal{H_Z}$ is infinite-dimensional, its Hilbert–Schmidt norm will be infinite, and thus it is not a Hilbert–Schmidt operator. Thank you for your suggestion; we will provide a more detailed explanation for this part.

[1] Candes E, et al. Panning for gold:‘model-X’knockoffs for high dimensional controlled variable selection, 2018

[2] Li et al. K-nearest-neighbor local sampling based conditional independence testing. 2024.