Multivariate Conformal Selection

On page 12, just before "The last equality is again by", $p\_{j}^\* \in \mathcal S\_{j}^\*$ should be replaced with $j \in \mathcal S\_{j}^*$... since the p-value corresponding to $j$ in $S^{\*}\_{j\to0}$ is 0.

A1: Thank you very much for pointing out the typo and the error in our proof. We have corrected the typo accordingly.

Let us clarify the corrected reasoning clearly. In the original manuscript, we aimed to show:

which is correct, yet we used a wrong path to prove this inequality. We claimed:

which is incorrect, since as pointed out, $\mathcal{S}\_{j \rightarrow 0}^{\*} := \mathcal{S}(p\_{1}^{(j)}, \dots, p\_{j-1}^{(j)}, 0, p\_{j+1}^{(j)}, \dots, p\_m^{(j)})$'s $j$-th argument is 0, not $p\_{j}^{\*}$. However, consider the following chain of inequalities:

This corrected argument allows us to properly continue the proof as originally intended. We have updated the manuscript to clearly reflect this correction.

In the proof of Theorem 3.5, could the authors comment on this sentence... but I was not able to understand it.

A2: Thank you for pointing this out. In fact, the assumption that $V\_1, \dots, V\_n, \widehat{V}\_{n+1}, \dots, \widehat{V}\_{n+m}$ have no ties is unnecessary. We can simply handle the case where $\widehat{V}\_{n+l} = \widehat{V}\_{n+j}$ by combining it directly with the scenario originally labeled (i), where $\widehat{V}\_{n+l} > \widehat{V}\_{n+j}$. After this adjustment, the proof proceeds without difficulty.

We have clarified this point and updated the manuscript accordingly.

In Section A.1, it would be good to provide a reference when it is stated: "By a standard result from conformal inference..."

A3: We added a reference (Vovk et al., 2005) to corroborate the claim.

Other Comments Or Suggestions

A4: We have corrected all the typos and addressed all the minor suggestions. We highly appreciate your careful review, which has helped us improve the clarity and quality of our manuscript.

It would be good to see an evaluation on classification outcomes as well.

A1: In our paper, we chose to focus primarily on regression tasks because they represent a more challenging setting for conformal selection.

In fact, the selection problem for classification (univariate or multivariate) can be directly reduced to the univariate conformal selection framework introduced by Jin and Candes (2023). This reduction explains why we did not include separate evaluations specifically for classification scenarios, as such evaluations would essentially revisit methods already established in the literature.

To clarify this point, let us briefly demonstrate the reduction explicitly:

• For the univariate classification setting, suppose the response space is composed of classes $\mathcal{Y} = \cup_{k=1}^K C_k$ with target region $R = \cup_{k=1}^{s} C_k$ (with $s < K$). Then, by defining a binary response: $\tilde{y}\_{i} = \boldsymbol{1}\\{y\_{i} \in R \\}$, the original selection problem directly translates into a univariate conformal selection problem, where we select samples with $\tilde{y}\_{i} = 1$.

• For multivariate classification case, e.g. suppose responses $\boldsymbol{y}\_{i}$ are drawn from a joint class space $\mathcal{Y} = (\mathcal{Y}^{(1)},\mathcal{Y}^{(2)}) = \cup_{k,\ell} (C^{(1)}\_k, C^{(2)}\_\ell)$, and the target region $R = \cup_{(k, \ell) \in \mathcal{I}}(C^{(1)}\_k, C^{(2)}\_\ell)$. Again, we define a binary indicator $\tilde{y}\_{i} = \boldsymbol{1}\\{\boldsymbol{y}\_{i} \in R \\},$ converting the original multivariate selection problem into a standard univariate selection task: $H_{0j}: \tilde{y}\_{n+j} < 0.5 \text{\quad versus \quad } H\_{1j}: \tilde{y}\_{n+j} \geq 0.5.$

Since $P(\tilde{y}\_{i} = 1)=P(\boldsymbol{y}\_{i} \in R)$, there is a direct correspondence between the multivariate and univariate nonconformity scores: $V(\boldsymbol{x},\boldsymbol{y}) = M\cdot\boldsymbol{1}\\{\boldsymbol{y}\in R\\} - \widehat{P}(\boldsymbol{y}\in R|\boldsymbol{x})$ and $V(\boldsymbol{x},\tilde{y}) = M\cdot\boldsymbol{1}\\{\tilde{y}\geq 0.5\\} - \tilde{\mu}(\boldsymbol{x})$ where $\tilde{\mu}(\boldsymbol{x})\equiv \widehat{P}(\tilde{y}=1|\boldsymbol{x})$. Moreover, regional monotonicity is simply the usual monotone condition of univariate conformal selection: $V(\boldsymbol{x},\tilde{y}=0)\leq V(\boldsymbol{x},\tilde{y}=1).$

Thus, every classification-based selection task can be naturally and effectively solved using existing univariate conformal selection methods. We will clarify this point explicitly in our final manuscript.

The following paper seems also related "Klein et al. (2025)".

A2: We have now cited this paper.

Figure 1 is too small to read properly.

A3: We have now increased the font size and improved the clarity of Figure 1.

Could you expand more about tie breaking? ...

A4: In our work, we follow the standard practice in conformal methods and break ties randomly. If instead, one used a deterministic worst-case tie-breaking rule-always ranking the test score behind tied calibration scores-the conformal p-value becomes: $p_j^{dtm} = \frac{1}{n+1}(1+\sum_{i=1}^n \boldsymbol{1}\\{V_i \leq \widehat{V}_{n+j}\\}).$ This rule ensures that $p_j^{dtm} \geq p_j$ for every test sample, making the test deterministic and conservative (though not uniformly distributed). Consequently, applying the BH procedure to these deterministic p-values yields a fully reproducible selection rule, but with a slight reduction in statistical power compared to random tie-breaking. We will clarify this trade-off explicitly in the final manuscript.

Footnote 2: The choice of $r_{n+j}$ needs to be exchangeable in which sense? ....

A5: In the original manuscript, footnote 2 mistakenly suggested that the choice of $r_{n+j}$ required some form of exchangeability. After careful reconsideration, we see clearly now that no such condition is needed. The point $r_{n+j}$ can be chosen arbitrarily from the region $R$. From the viewpoint of our proof, the critical step is the regional monotonicity condition. This condition alone ensures the relationship $p_{j}^\* \leq p_j$ under the null hypothesis. The oracle p-values $p_{j}^{*}$ are uniform precisely because of the exchangeability of the calibration data. The choice of $r_{n+j}$, therefore, does not influence the oracle p-value and imposes no extra exchangeability constraints. We have revised the manuscript accordingly to clearly state this simpler and accurate assumption.

How would one define regional monotonicity for a classification problem?

A6: Please see Q1.

In Theorem 4.1 why do you need to assume i.i.d.?...

A7: The proof of Theorem 4.1 indeed relies crucially on the strong law of large numbers. This result requires independence (or at least certain mixing conditions). Exchangeability alone is not sufficient here. Please see more detail in Appendix B.

The method requires an extra split... in higher dim.

A1: From our experience, the (multivariate) conformal selection procedure is insensitive to the size of calibration data - the calibration scores only affect the resolution of the p-values, which is typically sufficient when $|D\_{cal}| \geq 100$.

To support this point, we ran an additional experiment using $`mCS-learn`$ with real-data Task 2 at $q=0.5$. We set $|D\_{f-train}|=6400, |D\_{f-val}|=800$ and $|D\_{test}|=200$, respectively, and we consider different $|D'\_{cal}|$ shown below:

As shown, even with a small calibration set, the method maintains FDR control and strong power.

Why is a further split of the training set necessary...

A2: In short, this is because the computation of smoothed p-values require both calibration and test data; therefore to train $f_\theta$ based on the performance of a certain score, we would need to split the training set into two, which serve as calibration data and test data respectively.

What was the exact Algo used for soft ranking...typically very sensitive...explain

A3: We adopted the implementation in Blondel et al. (2020), with $\ell_2$ regularization and strength set to 0.1. Preliminary tests showed that within a reasonable range, different regularization strengths produce similar overall behavior in mCS. While cross-validation could optimize this parameter further, its exact value is not central to our primary contributions. To maintain clarity and simplicity, we chose not to include additional tuning steps for this hyperparameter.

The 2nd loss introduces another hyperparameter $\gamma$...

A4: This value was chosen based on a series of preliminary experiments. Similar to the previous question, while we could cross-validate on $\gamma$, for the simplicity of our presentation, we fix $\gamma$ to 0.5 in our main procedure.

In Algo 2, line 8, what is $k$...

A5: To accurately estimate the power of the selection rule given by $V^{\theta}$ (defined as an expectation in Eq. 2), we average empirical selection power over multiple random partitions of $D\_{f-val}$. Each random partition produces different power values, even with the same $V^{\theta}(x,y)$. We use $k=100$ partitions, which provides a stable and accurate approximation. We clarified this point in the updated version of our manuscript.

For the simulated study... bigger test set than 100.

A6: We appreciate the suggestion. While a test size of 100 may seem small for general machine learning tasks' evaluation, conformal selection performance is generally insensitive to test set size (as noted in Sec 3.1 of Jin and Candes, 2023). Additionally, we repeated our experiments over 100 independently generated datasets, ensuring the results are stable. This stability is also supported by our simulation results provided in response to subsequent comments.

Please report the std. errors...

A7: For both simulation and real-data experiments, each iteration involves sampling a new dataset, introducing variation in data and model training. Reporting std errors under these conditions would reflect primarily data variability rather than the inherent stability of the selection methods.

However, we agree numerical stability is important. Thus, in Appendix D, we provide an additional real-data experiment with a fixed dataset across iterations. Below are the observed std dev. of FDR and power (over 100 iterations) from this experiment:

FDR/Power:

These results confirm strong numerical stability for our proposed methods. Note that for $`mCS-learn`$, the trained model $f_\theta$ was fixed, so variability from model training is not considered here.

The choice of $r_{n+j}$...?

A8: We have added a detailed discussion about the choice of $r\_{n+j}$, please refer to our response under A5 for Reviewer 2 (PjuQ).

For both the generalized signed score and the clipped score, choosing $r_{n+j}$ on the boundary is already optimal. In this case, the first term $D_1$ is minimized (equal to 0), so no further optimization is needed to enhance selection power.

How ... to adapt when ... outputs a conditional distribution $\hat{P}(Y|X)$?

A9: When the model $\hat\mu$ outputs an estimated conditional distribution $\widehat{P}(y|x)$, the second term $D_2$ can be replaced by the predicted probability of being in the target region: $y \in R$, i.e. $\int \boldsymbol{1}\\{ y\in R\\} d\widehat{P}(y|x)$. This serves the same purpose: points with high predicted probability of satisfying the selection criterion will receive lower scores and are more likely to be selected.