Thank you very much for your very thorough dissection of our paper and insightful comments! We appreciate that you find our algorithm is “properly motivated”, with “potential to be very strong”. We hope to address your concerns and questions in the following.

Weaknesses

“I didn't see any discussion of why readers should think the proposed method would work [...]”.

The theoretical discussion in the supplement has some intuition of why the proposed algorithm will work. [...]

Thank you for highlighting this critical point. The underlying intuition is that given a base classifier model, the disagreement rate is a quantity that differs in and out of distribution only when the base model will underperform on a OOD batch. The reason we track this discrepancy is due to Definition 2, Line 949, 3rd inequality $err(h;Q_f) > err(h; P_f)$ where the algorithm’s calibration step is generating empirical values for the RHS, while the Deployment step compares an empirical estimate of the LHS to an empirical distribution of $err(h; P_f)$. If we believe Definition 2 is sound, then Idealized D3M works because it’s designed to empirically track D-PDD, while D3M is a crude approximation.

We agree that sampling over-approximates the set of interest. It’s likely that our sampling produces hypotheses that would’ve achieved higher error than the base classifier on the training set. Empirically, we find that although this may be the case, the strength of disagreement is more significant on deteriorating OOD data. Guarantees on significant disagreement gaps between ID and OOD despite sampling beyond the set of interest are currently beyond what the theory provides.

some of the theoretical development needs to be moved to the main text, as it's what ties the entire paper together [...]

We agree with your assessment of the importance of the theoretical binds. For this reason, we are editing in a theoretical insights subsection along with an informal definition of D-PDD as well as an informal lemma linking D3M to the tracking of D-PDD, while referring readers to the Appendix for a more formal treatment.

There are no experients that get at the heart of what's going on with the algorithm. I think synthetic experiments are an excellent opportunity to really probe what's going on behind the scenes of an algorithm [...]

Thank you for raising this point! We could demonstrate a proof of concept for the elevated OOD disagreement rates by the following experiment: we learn a posterior over decision boundaries on the synthetic Two Moons dataset in 2D. For ID and OOD samples, we illustrate how likely it is to sample a decision boundary that disagrees on those samples with respect to the mean decision boundary, building intuition for the discrepancy in disagreement rates.

The theoretical results hinge on assuming the model class is correctly specified, which is a major assumption that is rarely true in practice.

Indeed, this is an important limitation with regards to results. However, we encourage readers to view the theory as a perspective on how one might go about modeling the deteriorating OOD problem, as well as the insights insights which we derive and track directly in D3M (tracking $err(h;Q_f) > err(h;P_f)$, for instance).

Assumes that datapoints are available at once to make a decision of prediction on. Again, it's not clear to me how practical this setting is.

The paper seems to propose waiting to make PDD predictions until a large batch of test points has been accumulated. Why is this realistic? How would this work in time-sensitive situations?

An example of a practical setting would be D3M undergoing a silent trial at a hospital network, where patient data come in sequentially to fill up a batch before being processed with D3M. This is in contrast to paradigms of decision making where a decision is expected after an observation is recorded.

In Lemma A.1: was not defined. [...]

Our revised manuscript will provide the definition of $h$ following Definition 2.

Regarding $Q$ being a point mass, this violates the TV condition unless $P$ already concentrates nearly all of its mass on it. Our lemma explicitly assumes a bounded TV shift and a nontrivial disagreement gap, ensuring that the difference in disagreement cannot be solely attributed to small distributional changes.

We edited the supremum to be over $A \in \mathcal{F}$ of the underlying $\sigma$-algebra.

We respectfully note that the reverse direction (D-PDD → PDD) is indeed concluded: the final inequality shows that if the disagreement gap satisfies $err(f,Q_h​)−err(f,P_h​)\geq 2(\kappa+\epsilon)$, then it necessarily implies $err(f, Q_g​)> err(f, P_g​)$, which is precisely the PDD condition under $g=g’$.

Definition 4, 5, Eq (6)

Our manuscript has been revised with the suggested edits and clarifications.

Thm A.3 proves a bound on the FPR of the algorithm.

Towards better connecting the theory to the main paper

We appreciate the engagement with Theorem A.3. Indeed, the bound derived there is likely loose and primarily serves as a proof of concept—a theoretical guarantee that, in principle, our method can control the FPR under distribution shift. While the bound scales with the VC dimension of the hypothesis class and may be vacuous for large models, its main role is to motivate and justify the design of our algorithm. Empirically, we find that our method performs well with small batch sizes, suggesting that the theoretical sample complexity is conservative but directionally aligned with practice.

Experiments

The experiments are run in "{10,20,50}-shot scenarios" [...]

As per our discussion with Reviewer Ktyz, these are the sizes of the subset of data receiving a verdict from D3M. Importantly, methods in the literature typically give a verdict per point, whereas our setup’s goal is to give a verdict per batch or per data subset. We envision D3M as a monitor to a predictive model in deployment rather than a decision-making agent. Inputs on which the model predicts are bundled (one may imagine a moving-window of the last $m$ samples). A D3M flag would signify that a corrective action is in order, i.e. a re-calibration, a fine-tuning, a domain-adaptation modification, or even a potential retraining.

BBSD [...]

BBSD is indeed outperforming our model on the IM dataset for the age shift scenario. We suspect it is due to a strong covariate shift, where BBSD is known to perform really well. As for why other distance-based monitors such as MMD-D and Rel-MH aren’t achieving the same performance, we suspect they might be overfitting their representations to the training set.

Figure 2b [...]

Thanks for catching this! The y-axis is supposed to be labeled FPR@5%, our manuscript is since fixed. Distance-based learners are particularly susceptible to this type of FPR since they pick up on covariate shifts. We argue based on Figure 2a that this scenario is not a deteriorating shift, and therefore good monitors should resist flagging this change.

The x-axis in Fig 3 [...]

Thank you for the catch! These are the mixing coefficients between patients of different age groups as described in Lines 1315-1323. We will expand Fig3’s caption to be more comprehensive.

Overall Comments

Some references to specific Appendicies seem to be pointing to the wrong place

Thank you very much! We have proof-read our manuscript carefully.

"D3m does not trade off prediction accuracy" [...]

The classification accuracy is affected but only so slightly. Please refer to corresponding section of the response to Reviewer TFQc for a closer look.

I think the claim that "we answer all desiderata" it misleading [...]

Our empirical observations show that a low FPR is indeed achieved in practice. Intuitively, if the deployment sample came from the same training distribution, then its disagreement rate is beyond the $1-\alpha$ quantile with probability $\alpha$. In practice, FPRs may be slightly higher due to $\Phi$ being an empirical estimate and further approximation errors.

$\tau$

Thank you for pointing this out. We have included its definition.

Questions

How does the idealized algorithm that the theory is about relate to the actual algorithm? [...]

As per the above and as you have commented, the Bayesian sampling essentially over-approximates beyond the confines of the subset of hypotheses of interest. Another alternative is to try to optimize for $h$ directly via gradient descent with access to the training dataset in a similar fashion to Detectron (See TFQc). Users of D3M should view the theoretical results as qualitative guidance rather than quantitative guarantees, since the idealized D3M with idealized assumptions (mainly optimization oracle) do not fully hold in practice.

When will the algorithm fail? Are there any indicators or situations that users should look out for?

Empirically during testing, we found that when the temperature is high, while the ID FPR remains consistent empirically, the deteriorating OOD TPR is heavily impacted (See Response to Reviewer Ktyz).

Theoretically, a regime that our theory isn’t equipped to handle is Regime 2 (Line 1110), where we analyze a toy example and propose that well-trained base classifiers (low generalization error) are less likely to fall into this situation.

How can users reduce the variability [...]

We found that lowering the temperature below 1 helps reduce variance significantly at the cost of TPR, and are looking for hyperparameters and perhaps design changes that would rectify this.

Closing Remarks

Thank you once again for your helpful, attentive, and detailed feedback. We hope that our rebuttal addresses your questions and concerns, and we kindly ask that you consider a fresher evaluation of our paper if you are satisfied with our responses. We are also more than happy to answer any further questions that arise.

Thank you very much for the helpful comments! We are encouraged that you find our problem “very important in safety critical settings”, and that D3M is “competitive with prior work”. We hope to address your concerns and questions in the following.

Weaknesses

“D3M does not trade-off prediction accuracy.” [...]

Certainly! We hereby provide an ablation comparing the D3M base model (D3M) against the same feature extractor architecture coupled with a linear head (Lin), trained side-by-side for UCI Heart Disease, CIFAR-10, and Camelyon17, all else being equal. We report accuracy, F1, AUROC, and Matthew's correlation (for CIFAR-10, binary metrics are computed independently for each class and then averaging the results equally across all classes, without weighting by class frequency):

UCI (MLPModel)

CIFAR-10/10.1 (ConvModel)

Camelyon17 (ResNetModel)

We find that indeed, D3M trades off little to no prediction performance. We hypothesize further that this tradeoff is conditional on the generalizability of the representations used before the last layer, as MLPModels and ConvModels are trained from scratch, while the ResNet backbone in the Camelyon17 experiments are finetuned from the ImageNet models, although a lot more experiments are needed to validate this hypothesis.

“Potential for edge deployment.” [...]

We appreciate your observation regarding our claim about edge deployment. We agree that our discussion in this section was speculative and should have been more clearly framed as such. Our intent was to highlight that D3M’s source-free deployment mechanism offers practical advantages for local inference in e.g. hospital environments, where dedicated compute clusters may be unavailable or undesirable due to privacy and latency concerns. We acknowledge that further empirical study is needed to validate these claims, and we will revise the manuscript to reflect this more cautiously.

“While VBLL allows more efficient sampling compared to a fully Bayesian network, the price to pay is the lack of diversity of the sampled predictions.” To support this claim, I would expect to see a comparison of the diversity of predictions when using VBLL and a fully Bayesian network. [...]

How does the use of VBLL compare with other Bayesian approaches?

We found that without sampling once more from the logits in order to Monte-Carlo (MC) sample labels with which the base model can disagree with, the maximum disagreement rates achieved across $10000$ runs hover around 5% for both ID and deteriorating OOD samples, making them nearly indistinguishable by D3M. We found that sampling candidate disagreeing predictions from softmaxed and temperature-scaled logits rather than argmaxing allows us to bring the ID disagreement rates in $\Phi$ to around 30%-50%, where deteriorating OOD samples often score 5%-10% higher, making them distinguishable.

Under a rough comparison, for query size 100 and 5000 MC samples, fully Bayesian networks with logit argmaxing naturally achieve ~30% ID disagreement rate on the UCI task and ~37% ID disagreement rate on CIFAR-10. Importantly, while fully Bayesian models can reach comparable disagreement rates even without additional sampling tricks, they do so at significantly greater computational cost: D3M + VBLL trains $\Phi$ in 1 second while fully Bayesian D3M takes 17 minutes for UCI, for instance. In our setting, the goal is to highlight disagreement gaps between ID and deteriorating OOD data—not to maximize disagreement itself. Temperature scaling and softmax sampling already achieve this effectively, making added Bayesian complexity unnecessary.

For a more detailed discussion regarding this difference, please see our response to Reviewer Vf94. A complete report with comparisons of fully Bayesian networks to D3M + VBLL across all datasets across independent seeds is on the way and will be promptly added to a future revised version of the manuscript.

In Section 3.2, the authors claim that “D3M is significantly more scalable and practical for real-world deployment [than Detectron]”. [...]

Certainly! The following is a table listing the FLOPs at different computational stages of the algorithms. We assume a query size of 100, and the same hyperparameters as reported in the Appendix section.

Eventually, as the feature extractors grow in size, we notice that for D3M the cost is dominated by the training phase, whereas Detectron expends at least nearly the same compute during the calibration phase as its training phase. This is due to Detectron requiring finetuning on held-out subsets while D3M simply freezing the model in evaluation mode and sampling from the VBLL layers. Furthermore, gradient computations of Detectron need to be replicated on the deployment sample.

I’m quite confused by the Y-axis in Figure 2b. The label says TPR@5%, but this is in a setting with only a non-deteriorating shifts, so the true positive rate is undefined?

Our apologies for this mistake! In figure 2b, the label should be saying FPR@5%. We have adjusted the manuscript accordingly. Indeed, since this is a non-deteriorating shift (following Figure 2a), we want models to resist flagging. Covariate shifts are present as evidenced by their flagging from baseline naive covariate shift detectors.

The legend in Figure 2 and Figure 3 could be improved. I’m not sure what some of the abbreviations correspond to. For example, is DC the abbreviation for detectron? **

We have revised the manuscript to make the legend correspondences with baseline algorithms clearer. DC here refers to a Domain Classifier (per Rabanser et. al. 2019), i.e. a separate model trained on the same inputs to discern between ID and OOD data (this, of course, requires labeled OOD data). A detailed description of the Domain Classifier will be included in the Appendix.

Questions

I’m confused by the columns in Table 1. How can D3M be both deteriorating and non-deteriorating?

Indeed, these two columns refer to whether the algorithms 1. flags deteriorating shifts, and 2. resist flagging non-deteriorating shifts. Importantly, D3M achieves the latter, and provably so under certain theoretical conditions, detailed in Appendix A.

There is a duplicated “operating operating” in the first sentence of the abstract.

Thank you! This mistake is fixed.

“Underlines in our results indicate that D3M is close to the best baseline.” What is “close” here? It seems somewhat arbitrary given the differences in performance and nearly 10 points in some cases.

We wanted to use the underlines to highlight situations where most baselines severely underperform while D3M is nearing the performance of the top baseline, though as you highlighted, this is inconsistent with CIFAR-10, query size 20, where D3M gets outperformed by both Detectron and Deep Ensemble at roughly the same performance. For this reason, we have removed the underlines.

Please, clarify what exactly an N-shot scenario is. Also explain that the {10, 20, 50} in Table 2 correspond to shots in the caption.

Certainly! We appreciate you pointing out the lack of clarity. The N-shot scenarios correspond to the size of the unknown deployment dataset. Remote hospitals with scarcely an influx of patients would be more interested in the 10,20,50-deployment size scenarios, while major hospitals would already be enjoying high TPR from D3M.

Due to this confusion, as well as “few-shot” colloquially referring to single digit shots, we have henceforth decided to adopt the terminology of “query size” to refer to the quantity above, and will edit the manuscript accordingly as to further avoid misunderstandings.

Closing Comments:

Ablations reported in this rebuttal as well as other rebuttals will be published in a revised version of the manuscript.

Thank you once again for your helpful feedback. We hope that our rebuttal addresses your questions and concerns, and we kindly ask that you consider a fresher evaluation of our paper if you are satisfied with our responses. We are also more than happy to answer any further questions that arise.

Thank you very much for the helpful comments! We feel encouraged that you found our codebase as well as the introduction and the related work well structured! We are glad to hear that you found our algorithm clearly explained! In the following, we hope to address your questions and concerns.

Weaknesses

The abstract has two “operating”

Thank you! We edited the manuscript accordingly.

Evaluation on the UCI Heart Disease, CIFAR 10.1, and Camelyon 17 datasets can be more thorough with holdout participants or a dataset subset.

For all experiments, we construct D3M’s Phi training and ID splits by randomly splitting the original validation set. In each independent iterate of the experiment, random participants in the ID dataset as well as in the OOD dataset are selected from which run statistics contribute to the reported figures in the manuscript, thus each independent run “holds out” both ID and OOD participants. Could you perhaps mean a k-fold validation by splitting ID and OOD data into bundles and cross-validating TPRs? We are happy to run further experimentation to showcase the effectiveness of the framework if you could please elaborate on your suggestions on evaluation!

Temporal shifts are only shown for the UCI IM dataset with an artificial shift in the test data (i.e., it’s not evident in the original data). This compromises the motivation of the problem. I strongly suggest evaluating on another temporal dataset with an inherent temporal shift.

Our apologies for the inadequate clarity in the experiments’ descriptions. The IM dataset (name withheld) is a private electronic health records (EHR) dataset from a network of hospitals, comprising of static and dynamic features for patients’ hospital visits. This dataset is temporal by nature as data splits are arranged per half-years starting from 2017, where shifts in patient features become even more pronounced during the COVID period of 2020-2021. For the predictive task of 14-day mortality, our D3M base models trained on data before 2018 surprisingly do not underperform on later splits (Fig. 2a), therefore we make the assessment that this shift is non-deteriorating.

Furthermore, we also claim that an important feature of this dataset is its inherent temporal covariate shift, the evidence of which is in the elevated FPR of divergence/distance-based detectors of Deep Kernel MMD (MMD-D) and Relative Mahalanobis Distance (Rel-MH). Functionally, this means that patients’ static covariates as well as their labs, medications, and medical interventions have drifted over time, especially into the COVID years.

The impact on the target classification due to the VBLL layer is not clearly discussed

How does the VBLL layer impact classification accuracy?

Thank you for pointing this out. We hereby provide an ablation comparing the D3M base model (D3M) against the same feature extractor architecture coupled with a linear head (Lin), trained side-by-side for UCI Heart Disease, CIFAR-10, and Camelyon17, all else being equal. We report accuracy, F1, AUROC, and Matthew's correlation (for CIFAR-10, binary metrics are computed independently for each class and then averaging the results equally across all classes, without weighting by class frequency):

UCI (MLPModel)

CIFAR-10/10.1 (ConvModel)

Camelyon17 (ResNetModel)

We notice that side by side, $FE_\theta + VBLL_\theta$ is slightly worse than its linear head counterpart. We argue, however, that the sampling benefits provided by the VBLL layer enables the functioning of D3M. When compared to multiple heads (Deep Ensemble baseline) for deteriorating OOD detection, our method achieves noticeably higher few-shot TPR.

Please provide an ablation study on the parameters: temperature and alpha.

Certainly! The following table is the result of an ablation on the temperature parameter at query size $100$.

UCI

CIFAR-10/10.1

Camelyon17

By increasing the temperature, samples are more diverse and disagreements are more pronounced, evidenced by the elevated mean disagreement rates. This not only hurts OOD-TPR but also increases FPRs, as the “base ID disagreement rate” distribution is further skewed toward 1, making it harder to differentiate between ID disagreement rates and deteriorating OOD disagreement rates. The correct temperature balancing sampling diversity with reasonable ID disagreement rates (and thus ID disagreement thresholds) is thus a hyperparameter that we tune. In general, values between 1 and 5 are preferred.

We respectfully note that $\alpha$ is a user-defined significance level, akin to a tolerance in classical hypothesis testing. Thus, varying $\alpha$ does not meaningfully reflect the behavior or robustness of the algorithm itself, but rather tunes the system’s operating point: a higher $\alpha$ increases sensitivity but also false positives, while a lower $\alpha$ increases specificity. As such, varying $\alpha$ does not reflect the robustness of the method, but rather the practitioner’s risk tolerance — similar to choosing a 95% vs. 99% confidence level depending on deployment needs. We provide results at a reasonable default $\alpha=0.1$ and practitioners may adjust it depending on the criticality of their deployment context.

Questions

How does the VBLL layer impact classification accuracy?

Please see above.

How can the VBLL layer be extended to a regression problem?

We believe several works in the literature replace the categorical likelihood with a Gaussian likelihood, i.e. the VBLL layer would output a predictive distribution over real-valued targets—typically by learning a distribution over the regression weights and using Bayesian linear outputs to produce a mean and variance for the predicted value. The ELBO objective can then be adapted to use a Gaussian likelihood while retaining variational inference for the weights, allowing the model to capture epistemic uncertainty in regression.

In terms of deteriorating OOD detection for regression, we believe D3M has nice analogies with already-existing confidence region methods. We believe “the extent of disagreement” in classification is thus analogous to a $(1-\alpha)$ confidence region following a posterior distribution in Bayesian regression, perhaps reducing D3M to a sampling-based approach to OOD detection for regression.

Closing Comments:

Ablations reported in this rebuttal as well as other rebuttals will be published in a revised version of the manuscript.

Thank you once again for your valuable feedback. We hope that our rebuttal addresses your questions and concerns, and we kindly ask that you consider a fresher evaluation of our paper if you are satisfied with our responses. We are also more than happy to answer any further questions that arise.

Thank you so much for the helpful comments! We are encouraged that you find our writing clear and the paper well-structured. We hope to address your questions and concerns in the following.

Questions

Line 1: operating repeated twice

Thank you for pointing this out! We’ve removed the repeated word and proof-read carefully.

Can the authors provide the definitions of deteriorating and non-deteriorating shifts directly in the introduction?

We will qualitatively introduce post-deployment deterioration prior to discussing the key challenges with its monitoring, thank you for the suggestion!

Can the authors explain as to why measuring disagreement via a variational approach that too only for the last layer is a useful method for measuring disagreement? In my opinion, wouldn't there be limited diversity or limited choice of hypothesis class exploration if only the last layer is trained in that fashion?

In theory, the diversity should be severely limited. The ultimate key to the method is in fact a two-stage sampling. Each Monte Carlo (MC) realization corresponds to a set of logits sampled from the variational posterior.

From here, one may argmax the logits to get labels for each MC realization, for each sample in our batch. In our testing, we found that even with up to $10,000$ MC realizations, the sampled logits are not diverse enough to the extent that their argmaxes largely are the same. Disagreement rates hover at less than 5% on both ID and OOD batches, making them practically indistinguishable.

Thus, we sample once again from softmaxed, temperature-weighted logits of each MC realization in order to get more variability, allowing us to achieve 30%-50% ID disagreement rates across various experiments. Of course, all operations here are parallelized, making the computational costs acceptable, even comfortable.

With this in mind, consider now a fully Bayesian implementation. Indeed, even without the two-stage sampling above, we achieve acceptable ID disagreement rates. However, this comes at the cost of heavy computational overhead. One either reparametrizes layers, making each forward pass’s output essentially a sample of logits, requiring sequential forward passes to collect MC logits, or one must lift all weight tensors to include a sample dimension and ensure every downstream operation — including residual connections, batch norm, and other architectural components — is broadcast-compatible.

Importantly, while this is realizable, it offers no benefits to our two-stage sampling strategy. We find that when temperatures are tuned to yield ID disagreement rates in the 30% to 50% range, deteriorating OOD disagreement rates are easily discernable as on those inputs, D3M tends to disagree 5%-10% better, thus easily achieving high TPR.

In summary, the limited choice of hypothesis classes is adequately rectified by the double-sampling described above and in the manuscript, which we found to be empirically sufficient for the purposes of D3M’s 2nd and 3rd stages.

How do the results change (qualitative explanation is sufficient) if there are multiple classification heads instead of the variational training?

This is the architecture and design philosophy of Deep Ensemble adapted to monitoring model deterioration. We report in Table 2 that this monitor underperforms compared to other baseline methods as well as D3M. While leveraging head disagreements is a sensible insight, we believe D3M is more appropriate when unlabeled test data is scarce.

After T rounds, can you provide a graph of how the base model (ID) threshold changes? Is it more or less very similar to each other?

We report the ID thresholds for $\alpha=0.1$ across $10$ independent runs. Columns are the rounds $t \leq T=1000$ from which the $1-\alpha$ quantile is computed for $\Phi$.

The above shows that although some variability is exhibited for early $t \in T$, the threshold stabilizes with more independent rounds, as evidenced by the decrease in standard deviation. Importantly, even as early as 5 samples, the ID threshold is already stabilized.

In addition to metrics such as TPR, FPR, can the authors also evaluate a unifying metric such as Matthew's correlation coefficient in order to holistically characterize performance?

For tasks with known OOD deterioration we desire monitors to identically output 1 (positive flag). Therefore, the most important metric by which we compare detectors is the TPR. One may choose to view detection as a meta-learning problem wherein the sole input to the detector is the task, and the metric by which to assess the detector is how reliably it identifies OOD deterioration. The converse reasoning is true for FPR.

Regarding the base classifier’s performance however, we report their accuracy, F1, AUROC, Matthew’s Correlation, as well as the accuracy drop when testing on deteriorating OOD held-out data below. We additionally provide classification performance comparisons with a Linear head instead of a VBLL layer, all else being equal, trained side to side.

UCI (MLPModel)

CIFAR-10/10.1 (ConvModel)

Camelyon17 (ResNetModel)

The drops in OOD accuracy is symptomatic of model deterioration.

Can the authors also qualitatively compare their work to conformal prediction and risk control? Conformal prediction can also be an easy to adopt strategy to explain model failures and even identify distribution shifts.

Thank you for this suggestion! Conformal prediction (CP) methods typically rely on a held-out calibration set with labeled data to construct valid predictive intervals, making them challenging to apply in the fully unsupervised post-deployment setting we consider. However, in a lot of applied scenarios, labels do eventually become available either via explicit labeling or one of several outcomes realized within a said time period. The deployment-label free advantage of D3M wears off over time in a practical setting when compared to CP in this regard.

We also highlight that D3M is uniquely robust to non-deteriorating shifts that may still be flagged by CP methods due to covariate changes. While CP methods can signal increased uncertainty, they do not directly distinguish between the harmfulness of shifts or offer the same guarantees on FPRs. We believe that the scope and assumptions of D3M differ from those of standard CP and risk control approaches.

Closing remarks

The manuscript will be updated with all ablation results presented in this rebuttal.

Thank you once again for your helpful feedback. We hope that our rebuttal addresses your questions and concerns, and we kindly ask that you consider a fresher evaluation of our paper if you are satisfied with our responses. We are also more than happy to answer any further questions that arise.