Weakness 1: The methodological novelty is limited: the idea of adapting the calibration set with the selection strategy to retain exchangeability is not very novel. Although dealing with conformal prediction, [1] also constructs the reference set based on the selection rule. The investigated selection methods are also similar. Overall, the main contribution of this method may be the rigorous proof (which, due to the length, I was unable to verify in every detail within the time constraints).

To W1: Thanks for your insightful comments on the methodological novelty of our approach. We agree that recent research has explored selective p-values from various perspectives. However, our work differs significantly from these studies, and we would like to provide some clarifications.

• The references you mentioned have made valuable contributions by studying the properties of selective p-values, with a focus on individual p-values. However, our paper addresses multiple testing, which requires studying the interactions among all selective p-values rather than focusing on a single p-value. This distinguishes our work from previous works in the literature. Specially, in our selective setting, the p-values are complicatedly correlated and the selection is also data-dependent. This makes the validity of conventional BH procedure suspicious and requires rigorous verification.
• To address the difficulty arising from data-dependent selection, our main technical contribution lies in developing a unified analytical framework. This framework builds upon the conditional calibration framework [2], but it is not trivial due to the challenges imposed by selection in FDR control. The number of test units $|\hat{\mathcal{S}}_u|$ is a random variable that can be complicatedly dependent with the p-values. Even though we can replace $|\hat{\mathcal{S}}_u|$ with $m$ for valid FDR control, it would be too loose and lead to reduced power.
• To fix the selective randomness in analyzing FDR, we utilize the stability property of selection rule. Our stability condition helps mitigate the randomness in the test number which is not addressed in a single test. And dependence association between the selective p-values and the final rejection set can also be decoupled through the stability.

We hope that these interpretations can ease your doubts. If you have any further questions, please feel free to ask us.

[1] Jin, Ying, and Zhimei Ren. Confidence on the focal: Conformal prediction with selection-conditional coverage. arXiv, 2024.

[2] William Fithian and Lihua Lei. Conditional calibration for false discovery rate control under dependence. AOS, 2022.

Weakness 2: minor errors

To W2: Sorry for our incautiousness. Thank you for pointing out these errors. We will revise them in the future revision.

Questions: Is the idea of adapting the calibration set similar to constructing the reference set in [1]?

To Q: Your question is very thoughtful and we would like to discuss it with you in depth.

• Indeed, our adaptive strategy has the same intension as "swapped" strategy, that is to maintain the exchangeability between selected calibration set and test unit. From a theoretical standpoint, the selective p-values constructed by both strategies are valid p-values given the selection event.
• However, the core objectives of our adaptive strategy and the swapped rule in [1] and [2] are fundamentally different. Ours focuses on conducting multiple testing over these p-values and offering theoretical guarantees. In contrast, the p-values generated by the swapped rule are utilized to construct prediction intervals with selection conditional coverage, a property specific to individual cases and not reliant on interactions between these p-values.
• The motivation of our adaptive strategy is directly related to weak stability, where the selection rule satisfies $\mathbf{S}_{\mathcal{D}_c,\mathcal{D}_u}(X_i)=\mathbf{S} _{\mathcal{D}_c\cup\{Z_j\},\mathcal{D}_u\backslash\{Z_j\}}(X_i)$ for any $j\in\hat{\mathcal{S}}_u$ and $i\in\mathcal{U}$. Thus, the constructed p-values are based on the selection rule of $\mathbf{S} _{\mathcal{D}_c\cup\{Z_j\},\mathcal{D}_u\backslash\{Z_j\}}$ and this is different from the swapping selection rule $\mathbf{S} _{\mathcal{D}_c\backslash\{Z_i\}\cup\{Z_j\},\mathcal{D}_u\backslash\{Z_j\}\cup\{Z_i\}}$ .
• Moreover, the swapping selection rules are designed to satisfy a general range of selection scenarios rather than selection rules of specific properties. For weakly stable selection rule $\mathbf{S}$, employing the swapping selection rule$\mathbf{S}_{\mathcal{D}_c\backslash\{Z_i\}\cup\{Z_j\},\mathcal{D}_u\backslash\{Z_j\}\cup\{Z_i\}}$ can yield a selection result $\hat{S}_u^{-j}$ that differs from $\hat{S}_u$, unlike our method. This difference complicates the substitution of $\hat{S}_u^{-j}$ with $\hat{S}_u$to mitigate the randomness of the test number.
• From an empirical standpoint, our strategy is more computationally efficient since for each $j\in\hat{\mathcal{S}}_u$, we only need to compute the selection rule once, while the swapping rule needs $|\mathcal{C}_0|$times.

[1] Jin, Ying, and Zhimei Ren. Confidence on the focal: Conformal prediction with selection-conditional coverage. arXiv, 2024.

[2] Bao, Yajie, et al. CAS: A General Algorithm for Online Selective Conformal Prediction with FCR Control. arXiv, 2024.

Weakness 1: Comparison with self-consistent adjustment

Thank you for the valuable suggestion. We have incorporated theoretical and empirical comparisons with your proposed method. And these discussions will be added in the future version of our work.

• From the theoretical point of view, we have observed that the power loss associated with utilizing a selective conformal p-value is usually less than that incurred by the FDR-Linking method. To illustrate this, assume $\pi_0=0.7$ and $\alpha=0.1$ as in the simulation setting of quantile selection, then we derive $\alpha^{'}\approx0.025$. The AMT method adjusts the marginal p-value after selection by multiplying the selection proportion $\hat{\theta}=1/0.7$. This is equivalent to employing the BH procedure on the marginal p-value with $\alpha=0.07$, which evidently yields greater power than SCA. Additionally, AMT does not make full use of the information from the selection procedure. In contrast, our proposed method uses a smaller p-value than AMT, which suggests more power increase.
• In terms of empirical performance, as demonstrated in both cases from our paper, the SCA method suffers from a power loss, confirming our theoretical analysis. | | | | QUAN | | MEAN | | | --- | --- | --- | --- | --- | --- | --- | | | | FDR | Power | FDR | Power | | Case A | SCA | 3.59 | 88.7 | 2.85 | 84.7 | | | SPCV | 9.83 | 93.9 | 9.90 | 93.9 | | | AMT | 6.23 | 92.1 | 8.28 | 92.6 | | Case B | SCA | 3.79 | 75.7 | 3.38 | 66.4 | | | SPCV | 9.82 | 84.9 | 9.79 | 81.1 | | | AMT | 8.71 | 77.0 | 5.81 | 79.5 |

Weakness 2: Comparison with InfoSCOP

Thank you for the constructive comments and we apologize for overlooking this important reference. As you correctly point out, the InfoSCOP involves a procedure for FDR control after selection via applying BH procedure to selective conformal p-values, which aligns closely with the fundamental approach of our work. Notably, the FDR control guarantee in InfoSCOP requires the selection rule to satisfy a specific assumption, which can be transformed into the joint-exchangeable condition in our context. For strongly stable selections, our method can be simplified and degenerate into a form similar to InfoSCOP.

We would like to clarify the following key differences between our method and InfoSCOP:

• Firstly, we provide the FDR control guarantee for general selection rules with strong stability, which is beyond the joint-exchangeable selection. For example, the assumption in InfoSCOP is not satisfied by the quantile selection rule based solely on test data. Thus, their theoretical results are not applicable in such cases, while our framework bridges this theoretical gap.

• Secondly, our approach covers a wider range of selection rules. For instance, when dealing with weakly stable rules, we employ conditional calibration on adaptive p-values to ensure rigorous FDR guarantees. The table below compares the performance of our approach with InfoSCOP under mean selection rule. The InfoSCOP shows reasonable empirical performance, which is similar to ours. Therefore, it is possible that InfoSCOP may still work under mean selection, making it an interesting topic for theoretical investigation, which remains unexplored in InfoSCOP. In contrast, we provide FDR control guarantee under a variety of selection scenarios. | | Case A | | Case B | | | --- | --- | --- | --- | --- | | | FDR | Power | FDR | Power | | InfoSCOP | 9.85 | 94.0 | 9.80 | 78.4 | | Ours | 9.86 | 93.4 | 9.80 | 78.1 |

• Lastly, our approach and InfoSCOP are designed for different goals, resulting in different analytical frameworks. Ours is specifically designed to address the multiple testing problem across various selection rules. From the perspective of conditional calibration, our method is unified, where the BH procedure for strongly stable selection can be seen as a special case. As a comparison, InfoSCOP is an excellent work for selecting an informative set with FCR control, but it is not primarily designed for multiple testing after data-dependent selection. Their FDR guarantee is an extension of FCR control, which limits their method's applicability to different selection rules.

Based on your suggestions, we will include a comprehensive comparison of our method with InfoSCOP in future version. Hope the above interpretation can ease your doubts.

Question: Could you elaborate on why the conformal setup makes sense here?

Thank you for the insightful question. We would like to make some clarifications.

• Indeed, our method is versatile and not limited to conformal setups. For instance, in scenarios where only null-labeled data is available, such as in outlier detection, our approach can still generate selective conformal p-values and perform the appropriate procedures to control the FDR. In such cases, we assume that the distribution of calibration data is identical to the test data because the join-exchangeable selection rule is expected to be applicable in this scenario. However, it is not necessary for quantile or mean selection rules which based on test data only.
• Your comment on population level based selection is indeed insightful and valuable, particularly for scenarios where the goal is to directly select a subset from the original data. However, our framework offers a broader applicability beyond this. In many cases, we are only concerned about the specific selected subgroup, and the FDR over this set should be controlled. For example, in brain scan experiments, researchers hope to find brain locations for specific signal with FDR control. Given that there are several encephalic regions, each should be treated separately [1]. In such cases, controlling FDR within these subsets is of concern, and a global cutoff may have no theoretical guarantee.

[1] Efron, B., Simultaneous Inference: When Should Hypothesis Testing Problems Be Combined? AOAS, 2008.

Weaknesses for technical contribution: As introduced in the paper, the technical difficulty of sample selection among the selected units with FDR control lies in (1) constructing valid p-values in the presence of selection and (2) dealing with the dependency between p-values for FDR control. The current paper mainly addresses the first problem, while the second problem is only partially solved for quite restrictive selection rules. In fact, finding the exchangeable group/constructing valid conformal p-values for selected units has already been quite extensively discussed in Jin and Ren (2024) and Bao et.al (2024). It would be helpful to clarify the technical contribution given this context.

Thanks for your constructive comments on the technical contribution of our work. We acknowledge that our problem also relies on the idea of ``finding the exchangeable group/constructing valid conformal p-values for selected units'', which has been discussed in previous works [1] and [2]. However, our approach addresses a more challenging multiple testing problem. We would like to provide some clarifications to highlight the contribution of our work and explain how it differs from existing studies.

• First, our problem setup differs significantly from previous works, which focus on constructing prediction intervals after selection. The goal of this paper is to conduct multiple testing after data-dependent selection such that the final rejection set has controlled FDR in a finite sample regime. This process involves the complex interaction among the p-values and the data-dependent selection process. In contrast, [1] and [2] both proposed a swapped strategy to construct valid conformal p-values after selection and then use them to build prediction intervals with selection conditional coverage. The selection conditional coverage is an individual notion and only requires the validity of a single p-value. However, this does not account for the correlations among p-values and thus does not guarantee the validity of multiple testing procedures.
• Second, our main technical contribution lies in developing a unified analytical framework for handling the randomness arising from data-driven selection in the context of multiple testing. This framework builds upon the conditional calibration framework [4], but we go beyond it by tackling the challenges imposed by selection in FDR control. In our approach, the number of test units is denoted as $|\hat{\mathcal{S}}_u|$, which is a random variable that can have complicated dependencies with the p-values. Although it is possible to replace $|\hat{\mathcal{S}}_u|$ with a fixed value $m$for valid FDR control, doing so would be too conservative and would result in a significant loss of power.
• Finally, to address the data-dependent selection effects, we leverage the stability property of the selection rule. Through a detailed investigation of the stability properties, we demonstrate that our procedure can have finite sample FDR control across many important selection rules. Notably, this advantageous property of stability was not investigated in [1] and [2]. For example, under the mean selection rule, which we have confirmed to be weakly stable in our framework, [1] and [2] did not fully leverage their stability characteristics, leading them to adopt a different approach for constructing selective conformal p-values. Our approach is based on $\mathbf{S}_ {\mathcal{D}_ c\cup\{Z_ j\},\mathcal{D}_ u\backslash\{Z_j\}}$ in Section 3.3, while they are based on the swapped selection $\mathbf{S}_{\mathcal{D}_c\backslash\{Z_i\}\cup\{Z_j\},\mathcal{D}_u\backslash\{Z_j\}\cup\{Z_i\}}$. Our adaptive selective p-value is more computationally efficient as it only requires computing the selection rule once instead of $|\mathcal{C}_0|$ times for each test unit. Also, our p-value is related to the selected subset $\mathbf{S} _{\mathcal{D} _c\cup\{Z _j\},\mathcal{D} _u\backslash\{Z _j\}}$, which has a closer relation to our FDR control analysis.

We hope these clarifications are helpful. If you have any further questions, please feel free to reach out.

[1] Jin, Ying, and Zhimei Ren. Confidence on the focal: Conformal prediction with selection-conditional coverage. arXiv, 2024.

[2] Bao, Yajie, et al. CAS: A General Algorithm for Online Selective Conformal Prediction with FCR the Control. arXiv, 2024.

[3] Ying Jin and Emmanuel J Candès. Selection by prediction with conformal p-values. JMLR, 2023.

[4] William Fithian and Lihua Lei. Conditional calibration for false discovery rate control under dependence. AOS, 2022.

Weakness 1: the lack of references to Vovk's work in particular, having studied conformal p-values and conformal testing for a long time.

To W1: Thank you for the nice suggestions. We will incorporate more Vovk's work to enhance the clarity of the review. There are several literature we plan to review in the future version:

[1] Vovk, V., Gammerman, A. , and Saunders, C. . Machine-learning applications of algorithmic randomness. ICML, 1999.

[2] Papadopoulos, H., Proedrou, K., Vovk, V., & Gammerman, A. . Inductive confidence machines for regression. ECML,2002

[3] Vovk, V., Nouretdinov, I., & Gammerman, A. . Testing exchangeability on-line. ICML, 2003.

[4] Vovk V, Lindsay D, Nouretdinov I, et al. Mondrian confidence machine. Technical Report, 2003.

[5] Vovk V. Conditional validity of inductive conformal predictors. Machine Learning, 2013.

Weakness 2: experimental part is limited to a very classical testing setting in a conference where applications related to deep learning would be of interest

To W2: We greatly appreciate your feedback on the current experimental limitations, particularly the absence of a modern setting. Your insights have inspired us to explore the application related to deep learning.

We apply our method to the field of drug discovery [1] as an initial trial. Based on the DAVIS dataset [2], we aim to identify drug-target pairs with high log binding affinity (Y). Our hypothesis is $H_{0,t}: Y_t<9.21$ with target FDR $\alpha$=10%. To transform the structural information of protein and chemical compounds into numerical features, we attach a bidirectional recurrent neural network on top of the 1D CNN output to encode them. Subsequently, we train a small neural network with 3 layers and 5 epochs based on the encoded information $X$ and binding affinity $Y$.

Below, we present our basic results for the quantile selection rule. In this rule, j-th sample is selected if $\hat{\mu}(X_j)$ is larger than the 35%-quantile over the predicted values in the test set. The results are outlined as follows, demonstrating that our procedure (SCPV) can control the FDR precisely.

Due to limited time, more experiments related to deep learning will be made in future version.

[1] Ying Jin and Emmanuel J Candès. Selection by prediction with conformal p-values. JMLR, 2023.

[2] Mindy I Davis, et. al. Comprehensive analysis of kinase inhibitor selectivity. Nature Biotechnology, 2011.

Question: Could you compare or clarify the contribution with regards to the conditional calibration scheme mentioned in the article?

To Q: Your question is very meaningful. Here we make a detailed discussion about the contribution with regards to the conventional conditional calibration.

The conventional conditional calibration [1] offers a flexible framework to decouple the dependence between p-values. It requires a carefully constructed quantity $\Phi_j$, such that appropriate $c_i^*$ can be identified to satisfy the condition $\mathbb{E}[\mathbb{1}\{p_j\leq c^*_i\}/|\hat{\mathcal{R}}_j|\mid \Phi_j]\leq \alpha/m$, where $p_j$ is a p-value under null, $\hat{\mathcal{R}}_j$ is a substitution of the original rejection set $\hat{\mathcal{R}}$ and $m$ is the number of test units.

In our selective setting, the number of test units is $|\hat{\mathcal{S}}_u|$, which can be complicatedly dependent with both $p_j$ and $\hat{\mathcal{R}}_j$. And when analyzing the FDR, the event that j-th sample is selected is also involved. So our our primary focus is on ensuring $\mathbb{E}\left[\frac{\mathbb{1}\{p_j\leq c^*_i,j\in\hat{\mathcal{S}}_u\}}{|\hat{\mathcal{R}}_j|}|\hat{\mathcal{S}}_u|\mid \Phi_j\right]\leq \alpha.$ The conditional calibration framework primarily focuses on the correlation of p-values. However, a significant challenge arises because FDR control in a selective setting involves not only individual p-values but also the selection procedure itself. Consequently, the selective effects are unavoidable when implementing conditional calibration. To address this, we leverage the stability property of the selection rule, which allows us to effectively conduct analysis over the selected subset effectively and rigorously.

[1] William Fithian and Lihua Lei. Conditional calibration for false discovery rate control under dependence. AOS, 2022.

Limitation: The approach taken in the weak selection rule has a lower power.

To L: Thank you for pointing this out. The conditional calibration approach will lose certain power due to the pruning process. Our simulation results shown in the Appendix are based on deterministic pruning and it lacks power indeed. However, various techniques exist, which can enhance power through randomization. Here we present the improved results of heterogeneous random pruning, whose power is nearly as powerful as the BH procedure.