Regularized Maximum Mean Discrepancy for Variable Selection

Although the overall procedure seems to be sound, I have serious concerns regarding its theoretical foundation.

1. The authors proposed to solve Eq. (3), which is the objective combining empirical MMD statistics and $\ell_2$ regularization. Recall for least squares problem, $\ell_1$ regularization (lasso) helps with variable selection while $\ell_2$ does not. I believe the similar result applies here. So why does the author prefer $\ell_2$ instead of $\ell_1$ regularization?
2. For the evaluation of $\ell(\lambda)$ in Eq.(4) or Eq. (5), it involves the sample estimate of certain objective. So which type of data is utilized? The training data $\mathfrak{X}^{Tr}$ or testing data $\mathfrak{X}^{Te}$, or we need to perform training-validation split on $\mathfrak{X}^{Tr}$ and then utilize train-train data?
3. In the Algorithm the authors proposed to solve the optimization problem (3), which is unfortunately a non-convx problem in weight $**w**$. Then the authors resort to solve approximation optimization problem (6). What is the optimality gap between these? It is difficult for me to convince the effectiveness of this approximation algorithm.
4. Similarly, only the convergence analysis for solving the approximation optimization problem (6) is provided, but I cannot see the testing statistical power analysis when resorting to this approximation algorithm
5. Only the real data instead of the implementation code is provided for this paper.

• The term “Regularized MMD” in the title is somewhat ambiguous and does not fully capture the main idea of this paper.
• In [1], the simplex constraint yields a sparse estimator under the least squares setting, but it remains unclear if the solution to (2) demonstrates similar properties.
• The properties of $w_\lambda^*$ are not sufficiently investigated.
• Table 1: The statistical properties of both the original and accelerated methods should be included.
• The selection of benchmarked methods is limited [2-3]; there are numerous variants of (nonlinear) Lasso that could be considered.
• Type-I error studies should be integrated into the main text.
• An ablation study would help clarify whether using the ridge penalty is beneficial.

Reference

• [1] Meinshausen, Nicolai. "Sign-constrained least squares estimation for high-dimensional regression." (2013): 1607-1631.

• [2] Ravikumar, Pradeep, et al. "Sparse additive models." Journal of the Royal Statistical Society Series B: Statistical Methodology 71.5 (2009): 1009-1030.

• [3] Yamada, Makoto, et al. "High-dimensional feature selection by feature-wise kernelized lasso." Neural computation 26.1 (2014): 185-207.

My main concern is that, although the authors claim that weighted MMD is good for selecting variables, they do not provide theoretical guarantees that the optimal weight is beneficial for two-sample test or binary classification. Therefore the proposed method seems to be a little bit ungrounded.

• The literature review could benefit from a broader references. For instance, other parameter selection schemes, other variants of MMD objective etc.
• The presentation in sec 2.4 is somewhat unclear. Equation (4) and (5) have an intricate structure. Readers could benefit from a clearer presentation/explanation.
• In the experiments, only two standard comparing methods are considered. The experimental results could be strengthened by including more advanced baselines.
• In the variable selection algorithms, randomness is involved. E.g., random subsets are chosen to compute the MMD value. It is generally recommended to report the mean and standard error over multiple random trials.

• The theoretical results only concern the consistency and the convergence of the approximate algorithm. There is no generalization or approximation guarantee.

• This work can benefit from a more extensive empirical study. For two-sample test, only one baseline is compared, and the experiments were exclusively conducted on synthetic data. For classification, two baselines are tested, on synthetic data and one real-word data set. Moreover, there is no error-bar in the reported empirical results.

• Most crucially, while the proposed method is claimed to be a method of variable selection, there is little comparison to other variable selection methods, expect some empirical results o compare the false discovery rate and the classification performance with the Lasso method.