Typicalness-Aware Learning for Failure Detection

We thank the reviewer for the valuable comments and suggestions.

Q1：The definition of atypical samples is vague, making it difficult to accurately determine the typicality of samples, which may affect the method's effectiveness. Please consider providing some mathematical definitions and visual examples.

Thank you for your comments, and we have added the mathematical definition of typicality from the NeurIPS 2023 [ref1] in the revised version of our manuscript.

Furthermore, we present the visual examples in Fig. 7 (b). These examples may aid in clarifying the distinction between typical and atypical samples.

[ref1] Yuksekgonul, "Beyond Confidence: Reliable Models Should Also Consider Atypicality," NeurIPS 2023.

Q2：There is room for improvement in the specific implementation, such as typically calculation and dynamic magnitude generation, by exploring more advanced designs.

As depicted in Fig. 5 (b), we have conduced extra ablation experiments with K-nearest neighbor (KNN) distance and Gaussian Mixture Models (GMM) to assess typicality. These alternative measures did not enhance performance (lower AURC is preferable), thereby reinforcing the validity of adopting the mean and variance.

Moreover, it's important to note that KNN, GMM, and density-based methods may entail higher computational costs compared to our approach, given the high efficiency of utilizing mean and variance in the proposed method.

Thank you for your insightful feedback. Since the fundamental mean and variance calculation already yields satisfactory performance in our method, delving into more advanced designs holds promise for further enhancing our method in the future.

Q3：The paper only compare failure detection task; the impact of TAL on classification accuracy is not analyzed ( accuracy is noly provided in the table but not discussed).

Thank you for highlighting the importance of conducting a more thorough analysis of the impact of our method. In the revised version of our paper, we have incorporated the discussion on classification accuracy:

• We have analyzed the impact of TAL on classification accuracy across different datasets.
• We have discussed the relationship between improved failure detection and classification performance.
• We have compared these results with baseline methods.

This supplementary analysis will offer a more comprehensive view of the advantages of TAL, demonstrating its value not only for failure detection but also for the overall model performance.

Q4：Minor error: "pred" and "GT" in Fig.2 should be swapped.

We have corrected this typo in the revised manuscript.

Q5：In Section 3.2 "We empirically find this simple strately works well in our experiments." Please explain why the mean and variance can represent the features of a sample to measure typicality?

We selected the mean/variance based on insights from CORES (CVPR2024), indicating that ID samples exhibit greater magnitudes and variations in responses compared to OOD samples. Fig. 5 (a) visually represents the disparity in mean responses between ID (typical) and OOD (atypical) samples.

Moreover, Fig. 6 (b) depicts the correlation between typicality and density utilizing mean and variance, suggesting that typicality can serve as a substitute for density. Density, calculated by Gaussian KDE (Kernel Density Estimation), represents the likelihood of observing a data point within the distribution.

Q6：The confidence calibration methods aim to achieve more reliable confidence estimates, but fail in failure detection task. Could please explain that?

Thank you for raising this important question about the difference between confidence calibration and failure detection. This distinction is crucial for understanding the limitations of traditional confidence calibration methods in failure detection tasks.

Confidence calibration methods are designed to ensure that predicted confidence levels align with actual accuracy rates. For instance, in a perfectly calibrated model, out of 10 samples predicted with a confidence level of 0.9, we would expect 9 to be correct and 1 to be incorrect. While this alignment is crucial for calibration purposes, it may not suffice for effective failure detection.

In failure detection tasks, our objective differs. We aim for high-confidence predictions to be consistently accurate. The occurrence of even one inaccurate prediction among high-confidence samples poses a significant issue. This is due to:

• Failure detection requires the identification of all potential errors, including those within high-confidence predictions.

• In critical applications, it is imperative not to overlook any failures, irrespective of the overall calibration performance.

Our approach, TAL, mitigates this issue by enhancing the reliability of high-confidence predictions, particularly tailored for failure detection tasks. TAL enables us to enhance the detection of potential failures, even in scenarios where conventional calibration metrics indicate the model is well-calibrated.

Q7：The OoD data and ID data are presented in equal proportions. How does the ration of them affect the results.

Thank you for raising this concern. To tackle this issue, we have conducted experiments in three settings: new failure detection, out-of-distribution (OOD) detection, and old failure detection.

Our results indicate the following: TAL outperforms baseline methods in all three settings; For OOD detection, specialized methods perform better than TAL; However, most OOD-specific methods exhibit reduced effectiveness in general failure detection tasks.

The robust performance of TAL across all settings underscores its versatility as a solution for various failure detection scenarios. As for the data proportion, we observed that it does not affect our method much. A detailed analysis of these findings will be included in the revised manuscript.

Thank you for your valuable feedback on our paper.

Q1: Missing comparisons. A number of methods that should be compared against are missing. As CRL requires training, but SIRC doesn't, and given the length of the rebuttal period, I would be happy with a comparison with just SIRC.

The comparison results with SIRC are shown in Tab. 8 (see rebuttal.pdf), and we will add them to the final version. However, we are deeply sorry that we are unable to finish the results of CRL due to the limited time and resources during the rebuttal period.

Q2：No risk-coverage curves presented. Although scalar scores show overall failure detection performance, plotting out RC curves in my opinion tells a lot more about the performance of an approach.

The risk-coverage curves are shown in Fig. 7(a), and we will them to the main paper.

Q3：Odd ImageNet results. The accuracy for ResNet-50 on this benchmark is far below the standard accuracy achievable using cross entropy training (~76%) and standard augmentations. Thus I am yet unconvinced that this approach is able to scale up to more realistic data. CIFAR results may not generalise to real applications.

We apologize for the subpar ImageNet results in our initial submission. Due to time constraints, our focus was primarily on conducting comprehensive experiments on CIFAR, leading to insufficient training time allocated for ImageNet and subsequent underperformance.

To rectify this issue, we have conducted thorough experiments on ImageNet. The revised results are detailed in Tab. 8, showcasing the efficacy of our approach in scaling effectively to larger and more realistic datasets like ImageNet.

Q4：Missing references/attribution to existing work. There are a number of other published work that targets new failure detection that are not cited [2,3,4].

Thank you for your reminder. We have added these references in the revised manuscript.

Q5：Complexity of method. The actual method of TAL is quite complex, with a number of stages (d calculation, $\tau$ calculation), hyperparameters (Tmin/max, queue length) and ad hoc design choices (mean/variance calced along feature dimension, wasserstein distance, using $\tau$ to mix TAL and CE). This complexity limits ease of adoption, and makes it difficult to understand the reasons for TAL's efficacy. A more thorough ablation would greatly improve the understanding of the importance of each choice, aiding potential practitioners. Similarly, some intermediate experiments, (e.g. showing how error-rate changes according to "typicalness") would also make the method of TAL more convincing.

The following provides additional explanations and experiments to address your concerns:

• 1. Hyperparameters and Intermediate Variables: The hyper-parameters include queue length, Tmin, and Tmax, with ablation studies provided. The intermediate variable d is normalized within each batch to derive $\tau$, capturing the relative typicality during training.
• 2. Feature Representation: We selected the mean/variance based on insights from CORES (CVPR2024), indicating that ID samples exhibit greater magnitudes and variations in responses compared to OOD samples. Fig. 5 (a) visually represents the disparity in mean responses between ID (typical) and OOD (atypical) samples.
• 3. Ablation of Typicality Measures: As depicted in Fig. 5 (b), we have conduced extra ablation experiments with K-nearest neighbor (KNN) distance and Gaussian Mixture Models (GMM) to assess typicality. These alternative measures did not enhance performance (lower AURC is preferable), thereby reinforcing the validity of our selection of mean/variance criteria.
• 4. Reasons for TAL's effectiveness: Our approach tailors the optimization strategy for typical and atypical samples to alleviate overconfidence. In particular, for typical samples, our method prioritizes direction optimization by enhancing the TAL loss optimization with a large $\tau$. Conversely, for atypical samples, the small $\tau$ (i.e., large 1 - $\tau$) emphasizes the optimization of the CE loss that considers both direction and magnitude. This may prioritize the magnitude and reduce the impact of atypical samples on the direction, making direction a more reliable confidence indicator.
• 5. Error Rate vs. typicality: We have included an experiment illustrating the variation in error rates with typically (Fig. 6 (a)). This experiment offers valuable insights into the behavior of TAL.

Thank you and we will add the elaboration to the revision.

Q6：Missing details. It is unclear to me what confidence score is used in the end for TAL. Moreover, how is $\tau$ set during inference? Is it fixed or is it calculated in the same way as during training? What is the OOD dataset for ImageNet?

• Confidence Score and Inference: In the inference phase, TAL functions akin to a model trained with conventional cross-entropy. We calculate the cosine value of the predicted direction to derive the confidence score. This aligns with our focus on direction as a more dependable confidence metric, as highlighted in the introduction and the response to Q5-4 (Reasons for TAL's effectiveness).

• Calculation of $\tau$: $\tau$ is not used during inference, but only used during training for adjusting the magnitude/direction optimization according to the sample typicalness, as mentioned the response to Q5-4.

• OOD Dataset for ImageNet. We used the Textures dataset as the OOD dataset for ImageNet experiments.

Thank you and we will add the above clarifications to the final version.

We thank the reviewer for valuable suggestions and insights, which have significantly improved our manuscript.

Q1: Which version of SIRC did you use? SIRC is just a method to combine scores and it's unclear which scores are combined. In the original paper, the best-performing combination was Entropy+Residual for example. SIRC is also typically used with softmax scores, so when applied to TAL, is it used with the cosine measure or a softmax score?

In our experiments, we utilized the most effective variant of SIRC for the FD task, i.e. SIRC_MSP_z. This configuration combines Maximum Softmax Probability (MSP) with the feature. For the TAL+SIRC combination, we opted for a cosine measure paired with the feature.

Q2: Could you include the training recipe for the updated ImageNet results? Also, how are the OOD/ID datasets balanced i.e. what is the ratio between the number of incorrect ID predictions and OOD samples? The use of cosine distance as a confidence score is important to the story, and helps me understand the approach a lot better. I would suggest making this clear to the reader early on (e.g. in Figure 1). I would generally suggest editing the manuscript to provide a clearer story, i.e. incorporating some of the new plots into the method section. Hopefully the authors see the value in this.

• Training protocol for the updated ImageNet experiments: We employed a ResNet50 architecture as the base model. For data preprocessing, we implemented standard augmentation techniques. These include random cropping, horizontal flipping, and normalization. Our training configuration includes a batch size of 256 and an initial learning rate of 0.1. The learning rate was managed by a StepLR scheduler, which reduced it by a factor of 0.1 every 30 epochs. We chose SGD as our optimizer, with a momentum of 0.9 and weight decay of 0.0001. The total training duration was set to 90 epochs. To accelerate training, we utilized distributed training across four 3090 GPUs. For the TAL-specific parameters, we kept Tmax and Tmin unchanged. The queue length and its initial epochs were adjusted proportionally, following the same ratio as in CIFAR experiments. This approach ensures consistency across different datasets while accounting for the larger scale of ImageNet.

• Ratio of OOD/ID data: ID (correct+incorrect) : OOD = 1:1 . For baseline，incorrect ID: OOD = 451:1880 ; For TAL, incorrect ID: OOD = 445:1880.

• Thanks for the suggestion. In the new revision, we will revise Figure 1 for a more clear explanation of our method. We will explicitly illustrate the training and testing processes respectively. We believe this addition will effectively address the reviewer's question and improve the clarity of our paper.

Q3： How are the error/density plots calculated? On the training set at a particular snapshot? On the test set using the final queue?

Density, calculated using Gaussian KDE (Kernel Density Estimation), represents the likelihood of observing a data point within the distribution. The density is computed using the ImageNet test set. Typicality, denoted by $\tau$ in the original paper, is determined by calculating the distance of each test sample to the training samples, using a historical queue constructed from the mean and variance of the training set (recomputed by the last snapshot) to obtain typically.

Q4： FMFP appears to fail on ImageNet. I am not familiar with the method but it would be good to offer insight here, as well as caution that the method may suffer from poor scalability.

FMFP employs Stochastic Weight Averaging (SWA) and Sharpness-Aware Minimization (SAM). SWA and SAM can suffer from sensitive hyperparameters for different datasets. Since FMFP did not report experimental results on ImageNet, we directly transferred the hyperparameters from CIFAR to ImageNet. We speculate that FMFP might need to carefully adjust the hyperparameters on ImageNet to achieve sufficient accuracy. On the other hand, for our TAL method, we also directly transfer the hyperparameters from CIFAR to ImageNet and achieve significant improvements over baselines (shown in Table 8), which demonstrates the stability of our algorithm.

Table 1: The performance differences in Old FD task among various SIRC variants.

The Table shows the performance for all of the variants of the SIRC method on ImageNet. We observe that SIRC_MSP_z and SIRC_MSP_res achieve similar performance. Our TAL algorithm can significantly surpass the baselines. Thank the reviewer for the question.

Thank you for your valuable comments and kind words to our work.

Q1： in its current form the empirical quantitative evaluation is not fully convincing as it is bounded to computer vision tasks and models with very few comparison points on large scale datasets.

Thank you and the results on ImageNet are presented in Tab. 8 (see rebuttal.pdf).

It's worth noting that recent failure detection works, such as Openmix [CVPR2023] and FMFP [ECCV2022], primarily focused on CIFAR, and we followed the same setting for a fair comparison. However, the experiments on the large-scale ImageNet further demonstrate the effectiveness of the proposed TAL.

Q2： the authors insist on the intuition behind the added value with an explicit illustration in the motivation but I did not see an explicit evaluation of the typical vs atypical detection.

We would like to clarify that, TAL does not explicitly identify typical or atypical samples but dynamically adjusts the training optimization process for different samples by calculating a value that reflects the degree of typicality, in order to alleviate the overconfidence issue. In other words, TAL tailors the optimization strategy for typical and atypical samples.

Specifically, for typical samples, our method prioritizes direction optimization by enhancing the TAL loss optimization with a large $\tau$.

Q3：Could the authors derive a small set of atypical and typical data (labeled set) from a known ood dataset for Imagenet such as [1] and measure the ability of the proposed method to actually separate typical vs atypical data? i believe this work would benefit from such validation.

We have added a visualization of typical and atypical samples in the revised manuscript (as shown in Fig. 7 (b), which demonstrates the distinction between typical and atypical samples. Thank you for this valuable suggestion, and we believe this illustration may facilitate the understanding.

Conversely, for atypical samples, the small $\tau$ (i.e., large 1 - $\tau$) emphasizes the optimization of the CE loss that considers both direction and magnitude. This may prioritize the magnitude and reduce the impact of atypical samples on the direction, making direction a more reliable confidence indicator.

Thank you and we will add the above elaboration to the revision.

Q1： My question would be, then why can't we derive a typical vs atypical detection method from this value? If we can, it would be interesting to do so in order to empirically validate that the intuition behind the proposed method actually corresponds to what occurs during training.

Yes. Our $\tau$ can indeed be used to distinguish between typical and atypical samples, similar to other methods for assessing typicality (such as density). Density is calculated using Gaussian kernel density estimation (Gaussian KDE) and represents the likelihood of observing a particular data point within a distribution. Fig. 6(b) in our rebuttal.pdf provides a scatter plot showing the relationship between density and our calculated typicality with $\tau$, which demonstrates a positive correlation. The advantage of our method is that it consumes fewer resources and computes quickly, making it suitable for use during the training process. Thank the reviewer for the suggestion.

We thank the reviewer for the valuable comments and suggestions.

Q1: Most experiments were conducted on CIFAR. Results on ImageNet are inconclusive; only MSP was used as a baseline for comparison.

Thank you for the reviewers' comments. The question about ImageNet is a common concern among the reviewers. Due to character limits, the response to this important question can be found in the main rebuttal.

Q2: Experiments on ViT were conducted on CIFAR. It is not clear why CIFAR was chosen as a dataset for ViT instead of ImageNet.

We would like to humbly clarify that, previous works mainly conducted experiments on CIFAR and they did not report the results on Imagenet. Therefore, we adopted CIFAR for ViT for a fair comparison across all baselines. The experiments on ViT were made to demonstrate the model-agnostic nature of our method, showing its compatibility with both CNN and ViT architectures.

Q3：As SSL and fine-tuning pre-trained models become state-of-the-art, it is unclear if TAL method applies.

We conducted experiments on fine-tuning pre-trained models by freezing the feature extractor and only training the classifier. However, the results were unsatisfactory. We hypothesize this is because our method essentially avoids overfitting to atypical features. The frozen feature extractor limits the effectiveness of our method, as only the classifier layer doesn't have enough model capacity.

In the discussion period, we will continue exploring fine-tuning all layers of the network to see if our method can still be effective. Going further, we will also explore combining our method with other OOD detection algorithms.

Q4：Some parts are not clear: a. How are mean and variance calculated for features? b. How do you handle incorrect samples to update HFQ? c. How is $\tau$ and `d' are calculated? d. Variables are not explained, e.g., equation 3. Thank you for your detailed feedback. We will clarify each point in our revised version:

a. We calculate the mean and variance of each sample's feature channels based on insights from CORES (CVPR 2024). This approach stems from the observation that in-distribution samples show larger magnitudes (mean) and variations (variance) in convolutional responses across channels compared to OoD samples, which are a type of atypical sample. As shown in Fig. 5(a), The mean response of OOD samples is smaller than correct ID samples.

b. As stated in line 167 of the main paper, We update the Historical Feature Queue (HFQ) using a first-in-first-out method for correctly predicted samples. We discard the statistics of incorrectly predicted samples.

c. We will incorporate the following clarification to Eq. (7) in the revised version. For each new batch of samples, we calculate a distance `d' for each sample. We then normalize these distances within the batch (dmin and dmax represent the minimum and maximum distances in the batch). This normalization is crucial as typicalness is relative, and we use it to control the strength of optimization direction.

d. We will provide clearer explanations for all variables, including those in Eq. 3, in our revised version.

Q5：It is not clear how equation 10 works. The atypical sample $(1-\tau)$ will only be trained by CE loss, which is counterintuitive.

Our approach tailors the optimization strategy for typical and atypical samples to alleviate overconfidence.

In particular, for typical samples, we prioritize direction optimization by enhancing the TAL loss optimization with a large $\tau$.

Conversely, for atypical samples, the small $\tau$ (i.e., large 1 - $\tau$) emphasizes the optimization of the CE loss that considers both direction and magnitude. This may prioritize the magnitude and reduce the impact of atypical samples on the direction, making direction a more reliable confidence indicator.

Thank you and we will add the elaboration to the revision.

Q6：only use SVHN for OOD shifts; WILDS dataset [1] should be added.

As suggested by the reviewer, we have conducted additional experiments using the WILDS dataset. These results will be included in our revised paper. Due to the limited time, the result table will come in the discussion period.

Q7: Evaluation for semantic and covariate shifts independently.

The New-FD setting was introduced in an ICLR 2023 paper, and we incorporated it into our related work section to offer insight into recent advancements in the field. The experiments regarding semantic (OOD) and covariate shifts(Old FD) independently with the evaluation of New FD are shown in Table 8 of rebuttal.pdf.

Q8： 1.In Sec 4.2, which model did you use? 2.What are dynamic magnitudes? Why ? 3.how does T influence the direction? 4.Explain equation 10.

1. Model: We used ResNet50, and the training details were in the supplementary materials.

2. Dynamic Magnitudes: Dynamic magnitudes regulate the intensity of direction optimization. In contrast to LogitNorm, which employs a fixed logits vector magnitude, our approach adjusts T (magnitude) to modulate the loss function, as depicted in Equation 4. This enables us to dynamically control the optimization strength based on the typicalness of the sample.

3. Influence of T: T controls optimization strength to mitigate overconfidence. For instance, in a binary classification scenario with the logit direction [√3/2, 1/2], raising T causes the sigmoid output approaching to 1, thereby decreasing the cross-entropy (CE) loss. Consequently, higher T values lead to a less intense optimization of the logit direction.

4. Explain Equation 10: As mentioned in our earlier response to Q5, emphasizing the CE loss (not only relying on the CE loss, as it is regulated by $\tau$) for atypical samples enables the optimization of both direction and magnitude. This may help reduce the adverse effects of atypical samples on the direction, enhancing the reliability of direction as a confidence indicator.