Synthetic-powered predictive inference

Thank you for your thoughtful review and encouraging feedback. Below, we address your comments in detail.

1. The only question I had while reading this paper is the following. Theorem 3.5 provides a safety bound when P and Q are not close. While the authors acknowledge this potential discrepancy, their explicit lower bound on coverage when P and Q are not the same is described somewhat vaguely. In particular, there is no quantitative analysis showing, under some reasonable experimental settings, how the lower bound behaves for a fixed value of $m$ and different levels of discrepancy between $P$ and $Q$.

The bounds in Theorem 3.5 are ``distribution-free guardrail bounds" that do not depend on either $P$ or $Q$, or their level of discrepancy. In other words, the lower bound is for following quantity:

\inf_\text{all distribution P$and$Q} \mathbb{P}(\text{coverage event}),

and similarly, the upper bound is for the supremum of the coverage rates.

These bounds are determined solely by the sample sizes $N$, $m$, and levels $\alpha$, $\beta$, and are independent of any unknown distributions or observed data. Therefore, once $N$, $m$, $\alpha$, and $\beta$ are fixed, these bounds are constant and known. In fact, one can set $\beta$ in advance to control the lower bound on the coverage, as detailed in Algorithm 4 of the Appendix. In our experimental plots (e.g., Figures 1 and 2), these bounds are also shown (labeled as "Lower bound" and "Upper bound"), and we observe that, across different levels of synthetic data quality, the coverage of the SPPI method always lies within these bounds---as guaranteed by Theorem 3.5.

This is, in fact, one of the main advantages of SPPI: it provides a guardrail bound on the quality of inference. While high-quality synthetic data is certainly preferable, the method still yields meaningful inference even when the synthetic data is of low quality---which is important, given that we typically lack prior knowledge about the data quality.

2. Is there a specific analysis which shows the empirical behavior of lower bounds when P and Q show different levels of discrepancy?

Regarding the lower bounds in Theorem 3.5, we believe that our response to the previous comment, along with the experimental results in the manuscript, sufficiently addresses the question.

Theorem 3.3, on the other hand, provides coverage bounds that indeed depend on the discrepancy between the real and synthetic score distributions. The final theoretical bounds are given by the tighter of the two---that is, the maximum of the lower bounds and the minimum of the upper bounds from Theorems 3.3 and 3.5.

If the reviewer is referring to the bounds in Theorem 3.3, we kindly refer the reviewer to the new results presented in Item 1 in our response to Reviewer aUpi. There, the table presents both the bounds from Theorems 3.3 and 3.5---as well as the best of the two (denoted as ``effective bounds")---on synthetic data where the total variation distance between the real and synthetic score distributions is known. These results demonstrate how the bounds adapt to different levels of discrepancy between the real and synthetic score distributions: Theorem 3.3 dominates the bound when $P$ and $Q$ are close, whereas Theorem 3.5 dominates when $P$ and $Q$ are more distinct.

3. Since this algorithm has an extra hyperparameter beta that controls the window size, I’m curious about how sensitive the prediction sets are to different values of beta.

We agree that it would be interesting to examine the performance of our method under different choices of hyperparameters. We have conducted additional experiments to explore this, and we will include the relevant results in the revision (due to space limitation and the changed rules that do not allow pdf responses, we are only able to provide a summary here). We appreciate the suggestion.

At a theoretical level, the sensitivity depends on the similarity between the synthetic and real score distributions. Specifically, this can be seen in the score-transportation step: if the two score distributions differ substantially, then the transported score $T(S_{m+1})$ is likely to lie at one of the endpoints of the corresponding window---even under a small $\beta$. As a result, the coverage of the SPPI prediction set will reach the guardrail bound in Theorem 3.5, indicating that we gain little from the synthetic data.

On the other hand, if the synthetic scores closely resemble the real scores, the transported score is likely to lie within the corresponding window rather than in its endpoints---even for relatively large values of $\beta$. In this case, the SPPI prediction set resembles conformal prediction with a larger sample size, achieving near $1-\alpha$ coverage, and the effect of the hyperparameter becomes relatively minor.

We include experiments on the ImageNet data below. The table reports the performance of all methods across different values of $\beta$ for the Siberian husky and lighter classes. The results align with the behavior described above: for the lighter class, the synthetic scores closely resemble the real ones, and accordingly, the performance of SPPI remains roughly the same across different $\beta$ values. In contrast, for the Siberian husky class, where the synthetic scores deviate significantly from the real scores, the performance of SPPI closely aligns with the guardrail bounds. Note that this table also presents how the lower bound of Theorem 3.5 increases with $\beta$, becoming closer to the nominal $1-\alpha$ level.

Table: Results for the ImageNet data as a function of $\beta$
Conditional coverage (%) and prediction set size across 100 trials for different β values, for lighter and Siberian husky classes. The target coverage is 95%. Standard errors are in parentheses. See Figure 1 (main text) for additional setup details.

4. The idea in Section 3.4.1 about matching the synthetic scores depending on real scores deserves more than just a short remark. To me, using another model (e.g. least squares) to align synthetic calibration scores to match the distribution of real scores might raise concerns about overfitting when new test points come in. But the authors show that this doesn’t happen, which is surprising and kind of counterintuitive. I think it would be great if the paper could discuss this part in more detail.

Thank you for pointing this out; we will clarify this issue in the revised paper. The core concern---how a transformation that depends on the real scores can avoid overfitting and still provide valid inference---is precisely what is addressed by Theorem 3.5 and its extensions (Corollaries 3.6 and 3.7). These results show that the guardrail coverage bounds hold without any restrictions on the synthetic scores---they do not have to be i.i.d. and can even depend on the real data. Roughly speaking, this is because, in the derivation of the bounds, any effect of the synthetic data is eliminated by a deterministic component inherent in the structure of the SPPI procedure.

Therefore, at least in terms of maintaining the guardrail bounds, we are allowed to do ``anything”---including double dipping, overfitting, and other steps that would typically compromise statistical validity in other settings and procedures.

This theoretical flexibility allows users to adapt the synthetic scores to better align with the real scores in a data-driven manner, without compromising the validity of our guardrail coverage guarantees and without requiring data splitting. In our experiments (Figure 2 and Appendix I.2), we demonstrate that the K-nearest subset approach can improve performance by bringing the empirical coverage rate closer to $1-\alpha$. This method is motivated by prior work (e.g., ``Class-conditional conformal prediction with many classes'' by Ding et al.), which suggests that certain classes may have score distributions more similar to the class of interest. More generally, whenever there is reason to believe that a subset of the synthetic data---or a transformation thereof---better aligns with the real scores, it is worthwhile to apply such methods.

A closing remark

Thank you for the opportunity to clarify our theoretical results and for your thoughtful feedback on our method. If any part remains unclear, we would be happy to elaborate further. We sincerely believe our response addresses your concerns and will be helpful as you reconsider your evaluation.

Thank you for your thoughtful review. We're glad you found the paper easy to follow. It appears that your major concerns are related to high-level theoretical aspects of our method and the data-driven adaptation of the synthetic scores. We start by addressing these two concerns and then proceed by responding to the rest of your comments.

Q4: It might be helpful to include some high-level discussion on why the method could improve the efficiency and how the coverage is affected theoretically.

We thank the reviewer for this suggestion. As discussed in the introduction, the key question is how to leverage synthetic data---which can be abundantly generated through generative AI, for instance---to improve the sample efficiency of predictive inference in settings where data is scarce.

A naive approach of simply applying existing methods to the large synthetic dataset leverages the increased sample size, but coverage is not guaranteed due to the unreliability of the synthetic data. One might suggest evaluating the quality of the synthetic data in advance, but such evaluation is not feasible in the setting of interest, where real data is scarce.

The proposed method, SPPI, addresses this issue by providing distribution-free guardrail bounds, enabling the procedure to achieve both of the following properties:

• When $P \approx Q$, SPPI behaves like conformal prediction with a large sample size, improving efficiency and reducing coverage variability. This is essentially the result of Theorem 3.3.
• When the $Q$ doesn't approximate $P$ well, SPPI provably remains robust and guaranteed to yield prediction sets with coverage rate that falls within the theoretical guardrail bounds established in Theorem 3.5.

Our theoretical result is based on both Theorems 3.3 and 3.5, where the effective bounds are the tighter of the two in each case. Specifically, Theorem 3.3 provides coverage bounds that depend on the distance between the real and synthetic score distributions---the smaller this distance, the tighter the bound approaches the nominal $1-\alpha$ level. On the other hand, Theorem 3.5 provides bounds that are determined solely by the sample sizes $N,m$, and levels $\alpha,\beta$. In fact, one can set $\beta$ in advance to control the lower bound on the coverage, as detailed in Algorithm 4 of the Appendix.

In sum, SPPI does not require prior knowledge of how well $Q$ approximates $P$, and yet it automatically adapts to the quality of synthetic data, ensuring valid coverage.

Q6-A: In Section 3.4, the theoretical results are not very illuminating, since attaining the bounds in Theorem 3.5 really doesn't tell us anything about how the coverage is improved.

As noted above, Theorem 3.5 does not quantify the impact of synthetic data quality on coverage. Instead, it provides guardrail coverage guarantees that hold regardless of the synthetic score distribution.

Perhaps the reviewer refers to Theorem 3.3, as that characterizes how the coverage bounds depend on the similarity between the synthetic and real score distributions. In Section 3.4, we discuss strategies to improve the quality of synthetic data. When such strategies reduce the total variation distance between the real and modified synthetic score distributions, Theorem 3.3 implies that the empirical coverage will better follow the nominal $1-\alpha$ level.

Q6-B: Theory for the coverage/efficiency that is related to the quality of learning the synthetic score function

When data splitting is used to improve the quality of the synthetic scores, we already provide a variant of Theorem 3.3 that is formally stated in Appendix Section E.

For a more challenging setup, where synthetic scores are modified without data splitting, the real and synthetic scores are no longer independent, and thus Theorem 3.3 does not apply. Yet, the bounds of Theorem 3.5 continue to hold! In practice, we expect better empirical performance when the synthetic scores are better aligned---as we observe in our K-nearest subset experiments (Figure 2 and Appendix I.2).

Lastly, we kindly refer the reviewer to a related discussion on theoretical analysis for efficiency (i.e., prediction set size) in our response to Reviewer pG2H, Item 3.

Q5: Unclear why one might improve statistical efficiency by matching the distribution of real and synthetic scores. Any efforts to align the scores will be subject to the intrinsic $1/\sqrt{m}$ uncertainty in the real scores. Is this uncertainty still there, while you address the ``bias" due to insufficient data?

Yes, the intrinsic uncertainty still remains. The goal of the score transformation is to shift the synthetic score distribution $Q$ appropriately so that it better resembles the real score distribution $P$, and the ``uncertainty" in the real scores is part of the information they provide about $P$. Yet, in some cases, we can still benefit from the larger sample size of the transformed synthetic scores. This is evident by our K-nearest subsets experiments, which we elaborate on in Q8 below.

Q8: Why can selecting samples from different label classes help the prediction for a specific label class?

Our method relies on synthetic data only through their scores---i.e., the prediction error of the fitted model---not through the raw inputs or labels. The experimental setting is motivated by prior work (e.g., ``Class-conditional conformal prediction with many classes'' by Ding et al.), which suggests that certain classes may have score distributions more similar to the class of interest. Selecting a subset of such classes can help align the synthetic and real score distributions, potentially improving statistical efficiency.

Q1: In Theorem 3.3, $\varepsilon_{P,Q}^{m+1}$ depends on the TV distance of order statistics, which is a bit mysterious. How might this term depend on the TV distance between P and Q?

The following relation holds for the term $\varepsilon_{P,Q}^{m+1}$:

$\varepsilon_{P,Q}^{m+1} \leq \mathrm{d}_\text{TV}(P^{m+1},Q^{m+1}),$

where $P^{m+1}$ denotes the joint distribution of $m$ i.i.d. samples from $P$, and $Q^{m+1}$ is defined similarly. Thus, the quantity is upper bounded by the TV distance between the product distributions $P^{m+1}$ and $Q^{m+1}$. Although the right-hand side is simpler and more intuitive, we chose to include the $\varepsilon_{P,Q}^{m+1}$ term to present a tighter bound. We will add this discussion to the revised version.

Q2: The slack term $\beta$ is discussed in lines 167-172 that only takes effect in extreme scenarios. Is there a way to derive a bound for that extreme event to obtain a sharper analysis?

Thank you for pointing this out---we found that the statements in lines 167–172 are somewhat vague, and we will make the necessary revisions. We apologize for the confusion. Specifically, what the term $\beta$ accounts for is the case where test score $S_{m+1}$ is not covered by the corresponding window, rather than ``extreme” cases. Although the step where $\beta$ appears involves a tight inequality, the final bounds we obtain can be loose, as some of the proof steps apply the union bound---which we believe is inevitable for the goal of deriving distribution-free bounds.

In fact, we claim that the bounds provided in this work are already quite tight. We kindly refer the reviewer to Item 1 in our response to Reviewer aUpi for further discussion on this topic and new experimental results.

Q3: How should one interpret the quantities in Theorem 3.5? In particular, what does $R_j^+$ and $R_j^-$ quantify, intuitively?

Figure S3 in the appendix provides a concrete example of the real and synthetic scores along with their corresponding windows, illustrating how the guardrail bounds are derived. Specifically, $R_j^+$ and $R_j^-$ are ranks within the set of synthetic scores. The ranks $R_j^+$ and $R_j^-$ are chosen appropriately so that ``the $j$-th smallest real score is between $R_j^-$-th and $R_j^+$-th synthetic scores with sufficient probability" holds. Therefore, roughly speaking, the interval $[R_j^-, R_j^+]$ is a prediction interval for the rank of $S_{(j)}$ in the set of synthetic scores.

The lower bound in Theorem 3.5 represents the proportion of windows that entirely lie below the conformal threshold $\tilde{Q}_{1-\alpha}$ based on synthetic data---which is equivalent to the proportion of $R_j^+$ that are less than $\lceil(1-\alpha)(N+1)\rceil$.

Q7-A: Why do you choose beta=0.4 in the experiments? How would you recommend choosing the value of beta?

We choose $\beta = 0.4$ based on the guardrail bounds in Theorem 3.5, and we recommend selecting its value following the same principle. Specifically, given the sample sizes $m, N$ and the target level $\alpha$, Algorithm 4 in the appendix provides a way to set $\beta$ to meet a user-specified lower bound on coverage.

Q7-B: How do the results change with the choice of beta?

We kindly refer the reviewer to the discussion and new results provided in Item 3 of our response to Reviewer UUws.

Q9: m=15 is extremely small for Section 4.2. How would the results change with the value of m? How large N is compared with m to ensure that one gets some improvement?

Note that relevant discussions are provided in Sections I.1.1 and J.1, ``The effect of the real calibration set size,” along with supporting experiments---in case this was previously overlooked. There, we demonstrate meaningful improvements for larger values of $m$, such as 50. But, naturally, the degree of improvement from leveraging synthetic data tends to diminish as $m$ grows.

The amount of improvement depends not only on the synthetic size $N$ but also on the quality of the synthetic scores. If the synthetic score distribution is nearly identical to the real score distribution, then any $N > m$ will lead to improvement. However, if the synthetic scores are of low quality such that the procedure frequently hits the guardrail, the degree of improvement will be relatively minor, even with a large $N$.

Thank you for your thoughtful review. We are happy that you found the paper clear and enjoyable to read. It appears that your major concern is with the tightness of our theoretical results. We start by addressing this point and then proceed by responding to the rest of your comments.

1. The lower bound in Theorem 3.3 seems conservative as taking large values of beta compared to alpha in the experiments still yields good coverage.

We would like to argue that the bounds provided in this work are already quite tight. Our theoretical result is based on both Theorems 3.3 and 3.5, where the effective bounds are the tighter of the two in each case.

We first mention that the bounds in Theorems 3.3 and 3.5 are distribution-free uniform bounds---specifically, the lower bound in Theorem 3.3 essentially states the following:

\inf_{\text{all distributions P$and$Q}} \left[\mathbb{P}(Y_{m+1} \in \widehat{C}(X_{m+1})) - (1-\alpha-\beta+\varepsilon_{P,Q}^{m+1})\right] \geq 0.

This bound was constructed to control this infimum, and thus the ``conservativeness" should be discussed in terms of how close the above infimum is to zero. For specific distributions $P$ and $Q$, it is possible that the situation is better and we achieve a higher coverage.

In practice, one would prefer a tight bound that holds for any unknown data distribution. Such a uniformly tight bound is a challenging goal and is likely infeasible (how can we be both distribution-free and distribution-adaptive?). However, our second result---Theorem 3.5---essentially addresses this concern. Recall that Theorem 3.5 provides ``guardrail" bounds, which approach $1 - \alpha$ as $\beta$ increases (discussed in Section D.1 as well). Importantly, these bounds do not depend on the distributions of the real or synthetic scores. Further, the lower guardrail bound is tunable: for given sample sizes $m,N$, and target level $\alpha$, the hyperparameter $\beta$ can be set to meet the user-specified guardrail coverage bound (as detailed in Algorithm 4 in the appendix).

In fact, if $\beta$ is large, then---depending on the quality of the synthetic scores---the bound in Theorem 3.5 may dominate that of Theorem 3.3 and provide a tighter guarantee.

To illustrate this point, we ran new experiments with synthetic data where the total variation (TV) distance between the real and synthetic score distributions can be computed exactly. The table below presents the coverage rates for all methods, along with the theoretical bounds from Theorems 3.3 and 3.5, and the effective bounds in each case. We will include the new results, along with this discussion, in the revised version.

Table: Results on synthetic data.
Coverage rates across different TV distances between the real and synthetic score distributions, along with bounds from Theorems 3.3, and 3.5, and the effective bounds (minimum/maximum of the two). Standard errors are shown in parentheses. Results are shown for $1-\alpha=0.95$ and $\beta=0.05$.

2. I am slightly dubious about the assertion Line 167 "when P=Q [...] our procedure provides a tighter upper bound than split conformal prediction". If one has P=Q then there is no problem anymore to solve in the first place.

We are considering the case where the quality of the synthetic data is unknown in advance, as is often the case in practice. In particular, we do not know beforehand whether $Q$ equals $P$ or not.

However, we can still leverage synthetic data to improve inference without such prior knowledge, and that is precisely the goal of our work. We address this problem by proposing a procedure with guardrails---with the following properties (which hold despite us not knowing whether $P \approx Q$ or not!):

• When $P \approx Q$, SPPI behaves like conformal prediction with a large sample size, thereby improving upon the naive conformal prediction based solely on limited real data.

• When the $Q$ doesn't approximate $P$ well, SPPI still provides meaningful inference to some extent, supported by the distribution-free guardrail bounds established in Theorem 3.5.

These properties are also validated empirically: our experiments show that SPPI yields more efficient prediction sets when $P \approx Q$, and continues to provide valid inference when this approximation is poor.

3. There is no in-depth discussion about the error terms in the bounds of Theorem 3.3 and 3.5. In particular, it seems that the sum $\varepsilon+1/(N+1)$ is a bias-variance trade-off.

Thank you for pointing this out. We agree with the interpretation in terms of the bias-variance tradeoff. Specifically, the fundamental goal of this work is to develop a method that leverages synthetic data—which may follow a different distribution from the real data and thus introduce some bias—to improve the sample efficiency of the inference. One important point to note is that what is being traded for the bias is not only the variance. Increasing the sample size in conformal prediction has two effects:

• It reduces the variability of the prediction set: a larger sample size leads to smaller variance in the score bound $S_{(\lceil(1-\alpha)(n+1)\rceil)}$, which is a sample quantile of the calibration scores.

• It reduces the conservativeness of the prediction set: Recall that in split conformal prediction, a mass of $1/(n+1)$ is assigned to $+\infty$ to account for the unobserved test score. This is the source of the upper bound $1 - \alpha + 1/(n+1)$.

The proposed SPPI method allows for some bias from the synthetic data in order to gain advantages in both aspects.

4. Do authors have an idea of how the bias variance (mentioned above) scale in m,N even in simple cases and in particular whether there is some intuition in how to choose the scaling of N compared to m?

We would say that ``a larger $N$ is always better.” Specifically, regarding the `bias' term $\varepsilon_{P,Q}^{m+1}$, it does not depend on the synthetic sample size $N$, whereas the term $1/(N+1)$ decreases as $N$ grows. Thus, at least in terms of $N$, there is no tradeoff---a larger $N$ represents a strict improvement.

Moreover, the sample size of the real data, $m$, affects the granularity of the bounds in Theorem 3.5. These bounds can be interpreted as $1-\alpha -\epsilon_1$ and $1-\alpha+\epsilon_2$, where a larger $m$ allows for tighter lower worst-case bound (for an appropriate choice of $\beta$), thereby reducing the worst-case bias.

5. The framework is extremely general and could include weighted exchangeability, distributional shift. For these specific cases, does the algorithm proposed obtain similar guarantees to existing ones?

Thank you for this interesting question. Extending the method to the covariate shift setting is in fact not straightforward. Specifically, the SPPI method relies on a score transporter that maps each order statistic of the real scores to the space of synthetic scores. A key property used in the theoretical analysis is that the rank of the test score is independent of the (ordered) values of the scores. However, this independence no longer holds under covariate shift. We believe this presents an interesting direction for further investigation.

6. Could beta be correlated to the difference between P and Q ? Since the transporter makes the real and synthetic scores ``similar", I would assume that could be used to calibrate that step similarly to results for weighted exchangeability, distributional shift. In particular, could deriving Lemma 3.1 in the case where P is not equal Q (with additional error terms that illustrate the shift) provide tighter coverage bounds?

We appreciate this insightful question. If we can somehow construct a window satisfying Lemma 3.1 in the case of $P \neq Q$ (through some $(P,Q)$-dependent $\beta$), then yes---the SPPI procedure will achieve the bounds in Theorem 3.3, but without the $\varepsilon_{P,Q}^{m+1}$ error terms.

However, it seems that such a construction of the window could require knowledge of the exact distributions $P$ and $Q$, rather than merely a similarity measure between them. Moreover, our primary focus is on settings with a limited number of real data points, where accurately estimating $P$ or the similarity between $P$ and $Q$ a priori is not a reasonable target.

Notably, the weighted conformal method that works under general distribution shift (``Conformal Prediction Beyond Exchangeability'' by Barber et. al, mentioned by the reviewer) requires the weights to be set a priori and relies on the TV distance between the weighted score distributions to quantify the drop in coverage due to the shift. Consequently, it is unclear how the latter could tighten Theorem 3.3, especially in light of our Theorem 3.5 that allows users to adapt the synthetic scores to better align with the real scores in a data-driven manner, without compromising the validity of our guardrail coverage guarantees and without requiring data splitting.

A closing remark

Thank you for the opportunity to clarify why we argue that our theoretical analysis is quite tight, and for your thoughtful feedback on our method. If any part remains unclear, we would be happy to elaborate further. We sincerely believe our response addresses your concerns and will be helpful as you reconsider your evaluation.

Thank you for your thoughtful review. It appears that your major concern is with the robustness of our method to the quality of the synthetic data. We start by addressing this point and then proceed by responding to the rest of your comments.

1. How does the method perform with synthetic data from weaker generators or less relevant domains? Can the authors comment on robustness in such scenarios?

First, we would like to point out that our method relies on the synthetic data only through their scores---i.e., the prediction error of the fitted model---not through the raw inputs or labels. When the synthetic data are of low quality (i.e., the synthetic score distribution differs significantly from the real scores), SPPI provably remains robust and guaranteed to yield prediction sets with coverage rate that falls within the theoretical guardrail bounds established in Theorem 3.5. These bounds hold regardless of the quality of the synthetic data. Moreover, the lower guardrail bound is tunable: for given sample sizes $m,N$, and target level $\alpha$, the hyperparameter $\beta$ can be set to meet the user-specified guardrail coverage bound (as detailed in Algorithm 4 in the appendix).

At the same time, if the synthetic data are of high quality (i.e., the synthetic score distribution closely resembles the real scores), SPPI benefits from using the synthetic data. The empirical coverage is tightly regulated around the target level $1-\alpha$, with low variability. This aligns with the results in Theorem 3.3, as the total variation distance is small in this case.

Importantly, our method also allows for improving the alignment between synthetic and real scores using the real scores themselves, and without requiring any data-splitting. For instance, our K-nearest subset approach selects subsets of synthetic scores that are more aligned with the real ones to improve statistical efficiency. Crucially, such refinements do not compromise the coverage guarantees; indeed, we demonstrate in the K-nearest subset experiments (Figure 2 and Appendix I.2) that using this approach can improve performance.

2. Discuss or systematically analyze the impact of different types of synthetic data and score functions on efficiency and coverage—especially when using less realistic or lower-quality generators.

We thank the reviewer for the suggestion. Note that the experiments in the manuscript already include four types of synthetic data: for image classification, we consider images generated by Stable Diffusion, Flux, and images from different classes; for the MEPS dataset, we use past panel data.

We agree that further comparisons across different synthetic sources and score functions are valuable. To that end, we include in the table below new results for image classification using the HPS score function---complementing the APS-based results in the manuscript---evaluated across three different synthetic datasets. We observe a similar trend to the one observed in the manuscript.

Regarding the effect of the quality of synthetic data on performance, we kindly refer the reviewer to our response to Reviewer aUpi, Item 1, where we present new experiments with synthetic data for which the total variation distance between the real and synthetic score distributions can be computed. These experiments demonstrate how varying levels of the quality of synthetic data affect the theoretical coverage bounds, and also the empirical coverage is always within the effective bounds defined as the tighter of Theorems 3.3 and 3.5, as our theory suggests. We will include all the new results in the revised version.

Table: Results for the ImageNet data using the HPS score function
Marginal and per-class conditional coverage for three synthetic datasets. The target coverage level is 1 − α = 0.95. Other details are as in Figure 1 from the manuscript.

3. SPPI method does not guarantee that the prediction set (or interval) will always be smaller than (or even no larger than) the one constructed using only real data.

There are scenarios in which the SPPI prediction set is always shorter than the conformal prediction set based solely on real data. For example, when the real data size is very small---specifically, $m < 1/\alpha + 1$---the naive conformal prediction always yields a trivial prediction set, whereas SPPI can produce nontrivial, finite-width prediction sets.

But, more generally, we would like to note that while ``constructing a valid prediction set whose length (or more generally, measure) is almost surely shorter than that of the naive conformal prediction" may sound appealing, it may be very hard or even infeasible target in the distribution-free setting---especially when leveraging synthetic data whose distribution is unknown. The same holds even for weaker targets, such as achieving a smaller expected width. Intuitively, the reason is that the distribution of the synthetic data can range from very good to very bad, and so the only way to make sure that we do not harm the coverage guarantee of standard conformal prediction (which is already very tight) would be to ignore synthetic data. Our SPPI method sidesteps this challenge by introducing the guardrails, as explained in the paper.

Nevertheless, as discussed in Item 1, when the synthetic scores are aligned with the real ones, SPPI yields shorter, more efficient prediction sets. When the synthetic data are of low quality, the method remains safe, with coverage bounded by the worst-case guarantees of Theorem 3.5. These two properties are validated through our extensive experiments: in nearly all of our experimental results, the proposed procedure outperforms naive conformal prediction in terms of prediction set width, while still achieving reasonable coverage rates.

4. More practical guidance on choosing hyperparameters, diagnostics for misalignment, and post-hoc adjustment strategies.

We thank the reviewer for this suggestion, and we will add the following discussion in the revised version.

For the hyperparameter $\beta$, we recommend selecting its value based on the desired guardrail bounds, e.g., 90% coverage with the ideal target level $1-\alpha=95\%$. Specifically, Algorithm 4 provides a procedure to obtain such a $\beta$ for given values of $N$, $m$, $\alpha$, that meets the user-specified guardrail coverage bound.

Regarding the subset size for selection, we suggest following our experiments and fixing $N$ to be reasonably large (e.g., at least 300–500) and then selecting the $K$ nearest synthetic subsets.

As for diagnostics and post-hoc adjustment strategies, we first recall that only the distribution of the scores matters for our method, not the raw inputs or labels. Therefore, diagnostics should be done in the score space. To assess similarity between the real and synthetic scores, one can apply any 1-dimensional distributional tests such as the Cramér–von Mises or Kolmogorov–Smirnov statistics, or visually inspect the histograms of the real and synthetic scores. However, since $m$ is often relatively small, these diagnostics will be noisy but still useful to detect major deviations. Importantly, such diagnostics do not compromise the validity of our coverage guarantees, as discussed below.

5. Suggest practical ways to diagnose or quantify when the synthetic and real score distributions are too dissimilar for SPPI to be effective. Are there warning signs or automated checks practitioners could use?

Our main focus in this work is on the setting where the real data is limited and insufficient to provide useful inference---in such a setting, diagnostics based on the real data may be noisy. Importantly, our approach is supported with distribution-free guardrails: regardless of the quality of the synthetic data, the coverage rate always lies between some fixed coverage levels $1-\alpha - \varepsilon_1$ and $1-\alpha + \varepsilon_2$ (this is essentially the result of Theorem 3.5), where the lower coverage level can be pre-defined by the user. Put simply, there is no need for any ``warning signs.''

Nevertheless, one can apply the diagnostics described in the previous item to detect when the synthetic scores are far off from the real scores, indicating that the synthetic data is not useful for the inference task or could be modified to better align with the real data. In fact, the latter is precisely the approach we take in our K-nearest subsets method, which allows us to benefit from better-aligned synthetic data. As demonstrated in the paper, with this approach, the coverage rate becomes closer to $1-\alpha$ rather than hitting the guardrails.

6. For large-scale settings, how does the computational cost of SPPI compare to standard conformal prediction or other recent efficient conformal methods?

SPPI incurs almost no additional computational cost compared to standard conformal prediction at inference time. To see this, note that conformal prediction requires finding a sample quantile of the scores, while SPPI performs essentially the same computation three times. Specifically, the simplified formula for SPPI (Section 3.2) consists of three terms: $S_{(\tilde{R}^-)}$ and $S_{(\tilde{R}^+)}$---sample quantiles of the real scores---and $\tilde{Q}_{1-\alpha}'$, a sample quantile of the synthetic scores.