Robust Conformal Outlier Detection under Contaminated Reference Data

Thank you for the careful review and encouraging feedback. We answer your question below.

R1: How does the concept of contaminated reference set relate to the ambiguous ground truth case?

To the best of our understanding, the papers you referenced study ambiguity in the labeling process within a multi-class classification setting, where some examples may plausibly belong to multiple classes and this ambiguity is explicitly reflected in the labels. In contrast, our work focuses on outlier detection rather than classification, and our setting does not involve any explicit label ambiguity: all calibration points are labeled as inliers, but some may in fact be outliers. The contamination is entirely latent: we have no indication of which points are mislabeled, nor any signal of uncertainty in the labels.

That said, our results do offer some connection to the point of view of the papers you mentioned. As shown in Lemma 2.2 and supported by our experiments, when there is uncertainty about a point’s status, treating it as an inlier is a conservative strategy that preserves type-I error control. This highlights a potentially interesting connection between label ambiguity and reference set contamination, though the two settings at this point still appear quite fundamentally different in how they represent and handle possible labeling errors.

In any case, we agree this connection is worth exploring further in the future and will cite the papers you mentioned while incorporating this discussion into Section 5 of the revised manuscript.

Thank you for your thoughtful and constructive feedback. We appreciate the opportunity to provide some clarifications. To address your questions and strengthen the empirical foundation of our claims, we have also conducted a range of new experiments available at https://tinyurl.com/rcod-exps, which we reference below as Supp-Figure X and Supp-Table Y.

Our goal in this paper is to address a critical gap in conformal outlier detection: robustness to contamination in the reference set. To the best of our knowledge, no prior work offers a fully satisfactory solution to this problem. While our proposed method is deliberately simple, its theoretical guarantees are non-trivial, and our new experiments demonstrate its practical robustness.

R1: The method’s simple pipeline lacks deeper algorithmic novelty

We understand this concern, though we believe the intuitive nature of our approach is a practical strength. Moreover, while the method is intentionally simple, establishing its validity is technically non-trivial. Our analysis required developing a novel proof technique, which we believe offers theoretical and methodological value beyond the specific application studied in this paper.

R2: Assumes i.i.d. inliers

You're right that conformal prediction traditionally assumes i.i.d. inliers and a clean reference set. Our work relaxes the second of these assumptions: we allow the reference set to be contaminated with non-i.i.d. outliers. The assumption we retain is that the inliers themselves are i.i.d. We agree that relaxing the i.i.d. assumption on the inliers is an exciting direction for future work. However, doing so would likely require additional assumptions on the dependency structure, going outside the scope of this paper. We will clarify this point in the revision.

R3: Focus on low contamination ≤5%

Thank you for pointing this out. As described in our response to Reviewer 2tsw, we’ve extended our experiments to include higher contamination levels, and continue to observe consistent trends.

R4: Systematically test Label-Trim with alternative models

We appreciate the suggestion. In addition to Isolation Forest (tabular) and ReAct with a ResNet backbone (visual) used in the main manuscript, we’ve now added results for: Tabular data: Local Outlier Factor (LOF) and One-Class SVM (OCSVM) Visual data: ReAct with a VGG19 backbone and SCALE with a ResNet backbone As shown in Supp-Figures 2–3, Label-Trim consistently controls the type-I error across all models. As expected, when the outlier detection model is less effective (e.g., LOF, OCSVM), the candidate set becomes noisier and power decreases. Still, our method improves over the baseline. For the visual data, Supp-Tables 1–2 show results that closely mirror the trends in the main manuscript. We’ll include these in the revision.

R5: Random outlier injection oversimplifies real-world scenarios where outliers may strategically mimic inliers

We completely agree and have addressed this in our response to Reviewer 2tsw, where we design and test more challenging outlier scenarios.

R6: Real-world drift are untested

We share your interest in this important issue. In our response to Reviewer 2tsw, we present new experiments simulating drift in the outlier distribution over time, and find that Label-Trim remains robust.

R7: Explore adaptive labeling budgets instead of fixed m=50.

We refer you to our response to Reviewer 2tsw, where we discuss the role of the annotation budget m, and show that Label-Trim performs well across a range of values. We also highlight that its performance degrades gracefully as the budget decreases.

R8: How does Label-Trim perform on naturally contaminated datasets?

Please see our response to Reviewer 2tsw, where we explain why controlled contamination is necessary for rigorous evaluation and provide additional experiments modeling more realistic contamination.

R9: Would the method fail catastrophically when the number of outliers exceeds the labeling budget m?

No. Both our theoretical results and experiments show that type-I error control is preserved regardless of how small m is relative to the number of outliers. For instance, in Figure 3 of the main manuscript and Supp-Figure 1 (right panel), the number of outliers significantly exceeds the labeling budget, yet Label-Trim still controls the error and achieves meaningful power gains.

R10: The focus on low-error regimes (α=0.01-0.03) matches safety-critical applications but ignores moderate α

We’ve added experiments evaluating performance for higher type-I error thresholds. As shown in Supp-Figure 9, Label-Trim continues to perform well across all tested α levels. We'll incorporate these results into the revised version.

Thank you for your thoughtful and encouraging feedback. To fully address your comments, we have conducted additional experiments, available at https://tinyurl.com/rcod-exps. In our responses below, we refer to these as Supp-Figure X and Supp-Table Y.

R1: The effectiveness of the Label - Trim method depends on the accuracy of the pre-trained outlier detection model. The paper does not thoroughly explore how to select or improve the outlier detection model for better performance of the overall system.

You're absolutely right that the performance of Label-Trim depends on the quality of the outlier detection model used to rank points for annotation. To explore this further, and in response to your concern (as well as R4 from Reviewer Zuz4), we conducted new experiments using outlier detection models with lower performance than Isolation Forest, specifically Local Outlier Factor and one-class SVM on the tabular datasets. As shown in Supp-Figures 2–3, when the underlying model struggles to distinguish inliers from outliers, Label-Trim does not show meaningful power gains over the standard conformal method. This is expected, as the model’s scores no longer reliably identify outliers.

That said, as with any outlier detection task, we recommend using the best model available or fine-tuning a given model using a small set of labeled outliers, something that naturally improves performance. Importantly, Label-Trim is model-agnostic: it imposes no constraints on model complexity or architecture. This ensures that our approach remains compatible with future developments in outlier detection.

R2: Although the paper uses visualizations to present experimental results, there is a lack of visual aids to help readers understand the working mechanism of the Label - Trim method.

Thank you for this excellent suggestion. We agree that a visual explanation would help communicate the logic and steps of the method more clearly. In the revised version, we will include a new schematic illustration that walks through the Label-Trim pipeline.

R3: While the synthetic data experiments are useful, the authors could explore a wider range of outlier injection strategies.

We appreciate this point and want to clarify that all our experiments are conducted on real-world datasets (both inliers and outliers originate from real applications). That said, we agree that modeling more nuanced corruption is important. In our response to Reviewer 2tsw (R4), we include additional experiments with more strategic outlier injection strategies, which make the detection task more challenging.

R4: In real - world datasets, the preprocessing steps might be too simplistic.

We’re not entirely sure what specific aspect of preprocessing you are referring to. In our work, we follow the preprocessing protocols established in prior studies (Bates et al., 2023; Zhang et al., 2024; Yang et al., 2022). Our main departure is the introduction of contamination into the training and calibration sets to reflect realistic scenarios more closely. If there are particular concerns about the preprocessing that we overlooked, we would be happy to address them.

R5: There could be more diverse baselines considered.

We agree that comparisons with a broader set of baselines are useful. In addition to the four methods evaluated in the main manuscript, we have now added results for two additional outlier detection models on tabular data (Local Outlier Factor and one-class SVM), as well as two new models for visual data (ReAct with a VGG19 backbone and SCALE with a ResNet backbone). We’ve also included new experiments under higher contamination rates (up to 15%) and across a range of type-I error levels. As shown in the updated results, type-I error is consistently controlled, and power improves when the detection model separates inliers from outliers with reasonable accuracy.

R6: While type - I error rate and power are important metrics, additional metrics could be considered.

We focused on type-I error and power because they are the standard evaluation metrics in the conformal prediction and outlier detection literature. These metrics directly capture the validity (false positive control) and utility (detection rate) of the method. To provide additional insight, we also report the number of trimmed outliers in the contaminated reference set as a measure of cleaning effectiveness.

Thank you for your thoughtful review. We’ve conducted additional experiments to answer your questions and further support and clarify our contributions. These results are available at https://tinyurl.com/rcod-exps, and we refer to them as Supp-Figure X and Supp-Table Y in our responses below.

R1: In most experiments, m is fixed to 50. Generally, how should it be set to achieve good model performance?

We view the annotation budget m as primarily determined by resource constraints (how much it costs to label the data points) rather than as a tuning parameter that can be optimized by a general mathematical argument. This is because the power of our method depends not just on m, but also on the performance of the outlier detection model and the underlying contamination level, both of which are hard to characterize theoretically without much stronger assumptions than we make in this work.

That said, Figure 3 in the main manuscript provides interesting practical insight. In that experiment, the calibration set contains approximately 80 outliers. As shown in the figure, even a small annotation budget (e.g., m ≈ 20) is enough for Label-Trim to achieve power nearly matching that of the oracle conformal method that uses clean reference data. This suggests that even a modest budget (on the order of a quarter of the expected number of outliers) can already yield strong performance. Even more encouragingly, the power of Label-Trim appears to vary quite smoothly with the labeling budget in these experiments, indicating graceful performance degradation under tighter annotation constraints.

Thank you for giving us the opportunity to further highlight this important aspect of Figure 3. We will clarify this message and the associated practical guidance in the revised version.

R2: How does the proposed method perform when applied to real data that is inherently contaminated?

To answer this question, which also relates to Reviewer Zuz4’s comments, we conducted additional experiments simulating various realistic contamination scenarios. We summarize these results at the end of this response.

We also wish to clarify that all our existing experiments are already based on real-world datasets, meaning both the inliers and outliers originate from actual applications. Since our method relies on selective annotation under a limited budget, we simulate contamination in the reference set by injecting outliers. This setup reflects a realistic scenario where labeling resources are constrained and full manual annotation is impractical due to the need for domain expertise.

Moreover, a controlled contamination process is essential for evaluating our method because computing the performance metrics requires knowing the ground-truth labels. Specifically, to estimate type-I error, we need to know which test points are inliers, and to measure detection power, we need to know which test points are outliers. Summary of additional experiments:

• Varying contamination rate. We extend Figure 2 by increasing the contamination rate up to 15%. As shown in Supp-Figure 1, Label-Trim maintains type-I error control while continuing to outperform standard conformal and the “small clean” baseline in terms of power.
• Strategic outlier injection. Instead of injecting outliers at random, we selected outliers that resemble inliers—those falling below a given score percentile. Supp-Figure 5 (shuttle dataset) shows how these lower-percentile outliers increasingly resemble inliers, while Supp-Figure 4 demonstrates that Label-Trim still controls the error and improves power. Similar trends hold on the credit-card and KDDCup99 datasets (Supp-Figures 6–7).
• Test-time distribution drift. On the shuttle dataset, we simulate drift in the outlier distribution by contaminating the calibration set with high-percentile outliers and gradually shifting to harder, low-percentile outliers at test time. As shown in Supp-Figure 8, Label-Trim remains robust, maintaining error control and power throughout the distribution shift.

We will revise the paper to clarify how our experimental design reflects real-world constraints and to incorporate discussion of these new results demonstrating robustness to a range of contamination scenarios.