DeepDRK: Deep Dependency Regularized Knockoff for Feature Selection

Thank you for your insightful and constructive feedback on our manuscript. We appreciate the time and effort you invested in reviewing our work. We have carefully considered your comments and have made the following revisions to address the concerns raised above:

Weaknesses

• Though it enjoys theoretical result that a no free lunch situation for selection power when there is exact reconstructability in Appendix B, it seem that there is no theoretical guarantees on the power or explanations for that how the sliced-Wasserstein-based dependency regularization together with a novel perturbation technique introduced to reduce reconstructability can promote selection power.

ANSWER: To the best of our knowledge, theoretical analysis on FDR and power are mostly restricted to parametric designs. Without assuming the knowledge of the feature distribution (i.e. the distribution of $X$), how the SWC dependency regularization and the perturbation affect FDR (Type I error) and power (Type II error) is still an open problem, and we leave it for future research. Intuitively, adding independent components leads to less collinearity and thus improves power, at the cost that the swap property will be violated. Empirically, we observed that by enforcing swap property, the generated knockoff copies are often too close to the original design matrix, resulting in high collinearity so that although Type I error might be controlled, Type II error can be large. In such cases, we aim to sacrifice some FDR control for higher power in return. Experiments show that the SWC dependency regularization and the perturbation will result in slightly larger FDR (still relatively small), and much improved power (see Table 7 and Figure 7).

• It is not clear that how to enforce the swap property by the “multi-swapper” adversarial training procedure.

ANSWER: Thank you for pointing out this issue. We have added the description of the design of swappers and how they assist the knockoff transformer in achieving the swap property in the second paragraph of Appendix C. To repeat the part that involves the optimization: To generate the subset $B$, we consider drawing samples of $b_j$ from the corresponding $j$-th Gumbel-softmax random variable, and the subset $B$ is defined as $\\{j\in [p] ; b\_j=1\\}$. During optimization, we maximize Eq. (4) w.r.t. to the weights $\omega_i$ of the swapper $S_{\omega_i}$ such that the sampled indices, with which the swap is applied, lead to a higher $\text{SWD}$ in the objective (Eq. (4)). Minimizing this objective w.r.t. $\tilde{X}_\theta$ requires the knockoff to fight against the adversarial swaps. Therefore, the swap property is enforced. Compared to DDLK, the proposed DeepDRK utilizes multiple independent swappers.

We suggest checking Appendix C for the full description of how swappers are defined and used during this optimization.

• The motivation behind feature selection is high-dimensional data settings, which in my understanding means that the number of features is larger than the number of examples in the dataset. However, none of the simulated experiments include such scenario.

ANSWER: Thank you for pointing out this part. To the best of our knowledge, existing literature [1-5], both theoretical or methodological, concerns the classical linear regression setting when $n>p$ (or $n>2p$ in the knockoff framework as the knockoff introduces additional $p$ variables to the independent variables) . Extensions to the high dimensional setting are left to be unveiled.

We also want to mention that the knockoff generation part (e.g. the main objective of DeepDRK), as discussed in [1], can handle high dimensional data as there is no restriction on the generative model part in terms of the dimensionality. Hence, it is not a conflict of the statement in a high dimensional setting to the existing literature.

[1] Emmanuel Candes, Yingying Fan, Lucas Janson, and Jinchi Lv. Panning for gold:‘Model-X’ knockoffs for high dimensional controlled variable selection. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 80(3):551–577, 2018.

[2] Yaniv Romano, Matteo Sesia, and Emmanuel Candes. Deep knockoffs. Journal of the American Statistical Association, 115(532):1861–1872, 2020.

[3] James Jordon, Jinsung Yoon, and Mihaela van der Schaar. Knockoffgan: Generating knockoffs for feature selection using generative adversarial networks. In International Conference on Learning Representations, 2018.

[4] Shoaib Bin Masud, Matthew Werenski, James M Murphy, and Shuchin Aeron. Multivariate rank via entropic optimal transport: sample efficiency and generative modeling. arXiv preprint arXiv:2111.00043, 2021.

[5] Mukund Sudarshan, Wesley Tansey, and Rajesh Ranganath. Deep direct likelihood knockoffs. Advances in neural information processing systems, 33:5036–5046, 2020.

Thank you for your insightful and constructive feedback on our manuscript. We have carefully considered your comments and have made the following revisions to address the concerns raised above:

Weaknesses

• The loss in DeepDRK contains five terms, SWD, REx, cosine similarity w.r.t. swappers, SWC and the entry-wise decorrelation term. The necessity of introducing these five losses is under-explored in the paper.

ANSWER: Thank you for pointing out this issue. We conducted additional experiments for the ablation study section in Appendix L.3, and summarized them in the “Ablation Study Results” section of the general response above. Additionally, we want to point out that the SWD term is designed for imposing the swap property, which serves as a backbone of the model. Therefore, it cannot be removed as this would lead to complete deconstruction of the knockoff learning problem in our case.

• The necessity of DRP is unclear. There lacks comparison of $\tilde{X}\_{\theta}$ and $\tilde{X}\_{\theta}^{DRP}$ in empirical performance.

ANSWER: Thank you for pointing out this issue. We added one experiment to investigate the effect of $\alpha$ to the performance of FDR and power, where $\alpha$ controls the portion of the introduction of $X\_{\text{rp}}$, the perturbed $X$. Results can be found in Figure 7 in Appendix J of the revised manuscript. We verified that decreasing the value of $\alpha$ increases the power, however, at the cost of increasing the FDR. This is because the permutation inside $\tilde{X}^{\text{DRP}}\_\theta$ reduces reconstructability and destroys swap property simultaneously. And $\alpha = 1$ refers to the case without DRP (i.e. $\tilde{X}\_\theta$). Based on our result, we suggest choosing the value $\alpha$ between 0.4 and 0.5. And all results reported in our paper are based on the choice of $\alpha$ equals 0.5.

• It is clear that DeepDRK performs better than other baseline methods. But the reason has not been analyzed clearly and adding some intermediate results will be helpful. It is unclear how well the knockoffs generated by DeepDRK following the swap property and avoid overfitting compared to baseline methods.

ANSWER: Thank you for pointing out this part. We included a section "Appendix L.2" in the revised manuscript for measuring the swap property using three different metrics: mean discrepancy distance with the linear kernel and sliced Wasserstein 1 & 2 distances. This reveals the relationship between the sample level swap property and the performance in feature selection. We summarize this part in the “Measurement on Swap Property'' of the general response above.

Besides the verification on the swap property, we also clarified that we avoid overfitting via early stopping in the revised manuscript. Details can be found in Appendix I, which is a newly added section that includes the training algorithm of the knockoff transformer. Its effectiveness is proved given the results in Table 3 & 4, compared to other benchmarking models.

• The results w.r.t. the Gaussian mixture seems inconsistent with that in the original DDLK paper (DDLK performs the worst in this paper while it performs better than deep knockoffs and knockoffgan in the original paper).

ANSWER: Thank you for pointing out this part. However, we want to point out that the Gaussian mixture experiment considered in our paper is different than that in the original DDLK paper as we considered a lower signal strength (i.e. $\frac{p}{15\cdot \sqrt{n}}\cdot \text{Rademacher(0.5)}$ or $\frac{p}{12.5\cdot \sqrt{n}}\cdot \text{Rademacher(0.5)}$ for the distribution of $\beta$). The original DDLK uses $\frac{p}{1 \cdot \sqrt{n}}\cdot \text{Rademacher(0.5)}$ as the signal strength, which is larger than ours. The reason we chose the lower signal strength cases is because we found all models perform similarly well on the original case. Descriptions and details concerning this experiment setup can be found in the first paragraph of section 4.3.

Additionally, we also want to mention that in the references [1, 2] listed below, the authors independently identified the underperformance of DDLK on some Mixture of Gaussian data and on some real datasets. Therefore, our results in this paper are not inconsistent with the current literature.

[1] Derek Hansen, Brian Manzo, and Jeffrey Regier. Normalizing flows for knockoff-free controlled feature selection. Advances in Neural Information Processing Systems, 35:16125–16137, 2022.

[2] Shoaib Bin Masud, Matthew Werenski, James M Murphy, and Shuchin Aeron. Multivariate rank via entropic optimal transport: sample efficiency and generative modeling. arXiv preprint arXiv:2111.00043, 2021.

Thank you for your insightful and constructive feedback on our manuscript. We appreciate the time and effort you invested in reviewing our work. We have carefully considered your comments and have made the following revisions to address the concerns raised above:

Weaknesses

• Some arguments of the proposed method are not validated with corresponding experimental results. For example, “multi-swapper” is used to better achieve swap properties. But there are no experiments to justify how swap property changes when changing from single swapper to multi-swapper. I think authors should also introduce how to empirically measure the swap property. Since the proposed method relies on regularization to enforce the swap property, which is not guaranteed by design.

ANSWER: Thank you for pointing out. We included the ablation study on all the regularization terms in the Appendix L.3 of the revised manuscript. Please also see the “Ablation Study Results” section in the general response above for the complete description of the revised ablation study. We also included the section of Appendix L.2. for measuring the swap property. The summary of this experiment can be found in the “Measurement on Swap Property” section of the general response above.

• The proposed method uses many regularization terms. Some of the regularization terms have ablation studies, but others are not. For example, L_swapper and L_ED are not included. The effect of $\alpha$ in Eq.~(9) is also not investigated. Moreover, there are four hyperparameters that require tuning, making the proposed method hard to tune.

ANSWER: Thank you for pointing out this. As mentioned previously, we completed additional ablation studies considering more terms proposed in this paper, which includes the study of both $\mathcal{L}\_\text{swapper}$ and $\mathcal{L}\_\text{ED}$. Results can be found in Appendix L.3 in the revised manuscript. And the summary can be found in the "Ablation Study Results" section of the general response above. Based on the ablation study, we verified that $\mathcal{L}\_\text{ED}$ only affects the model’s training stability, rather than its feature selection performance. As a result, we suggest removing this term if no instability issue is observed during training. And the resulting number of tuning hyperparameters during training is identical with KnockoffGAN.

Besides the ablation study, we also investigated the effect of $\alpha$ on the performance of FDR and power. Results can be found in Figure 7 in Appendix J of the revised manuscript. We verified that decreasing the value of $\alpha$ increases the power, however, at the cost of increasing the FDR. This is because the permutation inside $\tilde{X}^{\text{DRP}}\_\theta$ reduces reconstructability and deteriorates swap property simultaneously. We also want to point out that this hyperparameter appears after training, which can be adjusted without introducing any training burden.

• The regularization terms largely come from existing papers; I think authors should better justify what is their contribution on top of existing papers.

ANSWER: Thank you for bringing up this point. We added one paragraph to the last part of section 3.1.1 to summarize how these terms collaboratively contribute to the guarantee of swap property. We want to point out that these terms are first used in knockoff generation for feature generation in this paper, which improves the effectiveness of the generation of knockoffs at sample level. And the utilization of these regularization terms is only one part of the contribution of this paper. We additionally summarized the overall contribution of our paper in the general response above in threefold.

Questions

• Since most dataset is not very large, but the model size is quite large. Did authors try to change the model size to see how it impacts the performance? Maybe the model could be smaller.

ANSWER: Thank you for bringing up this point about the model size effect. We prepared new experiments and investigated the effect of the model size (in different hidden dimensions and in different numbers of layers) to the performance in FDR and power. Results can be found in Table 3 & 4 in Appendix K. We observed slightly reduced power when considering smaller models in the number of layers and in the number of hidden dimensions. However, in many cases, since we observed similar performance between DeepDRK (e.g. the current setup) and the reduced models, we think the model size is not a strong influence factor to the feature selection performance. Besides, we also applied early stopping during the model training (stated in Appendix I, a newly added section that includes the training algorithm), which is an effective approach (given the results in Table 3 & 4) to prevent potential overfitting issues. This should also mitigate the effect of the model size to the performance of the model.

Thanks a lot for your appreciation of this work. We further improved the manuscript based on reviewers’ comments and suggestions. Please see the general response to all reviewers for details.

Measurement on Swap Property

Equipped with all above losses, in Appendix L.2 we compare the swap property of DeepDRK knockoff and baseline methods using different metrics as swap loss, namely mean discrepancy distance with the linear kernel, and the sliced-Wasserstein distances. The results can be found in Table 5. We observed that, in most cases, DeepDRK achieves the smallest values on the swap loss compared to other benchmarking models. This explains why DeepDRK outperforms the other models on FDR. We also found out that in the case with the mixture of Gaussian data, the swap losses for DeepDRK are not the smallest compared to KnockoffGAN, yet DeepDRK is the only model that controls the FDR and yields relatively high power. This suggests that DeepDRK successfully trades in a small fraction of FDR for considerable power improvement. Supporting evidence can also be found in Figure 7 of Appendix J, where the effect of the perturbation coefficient is studied.

Summary of the Contribution

We provide an outline of the contribution of this work in threefold: First, the competing nature between the swap property and reconstructability in knockoffs are identified, which turns out to be crucial when designing (non-parametric) knockoff algorithms for feature selection. Second, to address the issue, we propose the DeepDRK pipeline for generating knockoff copies with novel dependency regularizations. We use experimental studies to showcase the necessity for each term. Third, we demonstrate the effectiveness of DeepDRK on synthetic, semi-synthetic and real-world datasets. By comparing to benchmarking models, we show that DeepDRK generates high quality knockoffs, such that the feature selection across various datasets achieves high power and low FDR.