Optimizing Calibration by Gaining Aware of Prediction Correctness

Q12. Can our method be used after a train-time calibration method? Would like to see the complementary strengths

In this paper, we focus on post-hoc calibration, assuming that the classification model weights remain unchanged. However, we also explore potential complementary strengths between train-time calibration methods and our approach. To this end, we combined our CA loss with the loss in MDCA or DCA to train an image transformation-based calibrator. The results are presented below:

ImageNet-A:

ImageNet-R:

ImageNet-S:

ObjectNet:

We cannot observe complementary strengths of MDCA loss or DCA loss to CA loss. This suggests that some training-time calibration methods may not directly benefit the post-hoc calibration system. As discussed in Q5, they may also face the "narrowly wrong prediction issue." A similar result can also be found in Table 1, where CA + CE did not bring further improvement.

Q7. Is the post-hoc calibrator capable of calibrating non-ground truth classes as well?

Yes, it does indirectly affect the confidence distribution of non-ground truth classes as well. Our method applies a temperature to the logits (pre-softmax activations) for all classes. This operation changes the relative spread of the logits for all classes, not just the ground truth. As a result, the probabilities of non-ground truth classes are also calibrated in relation to the predicted class.

Q8. What is the performance of the method under the SCE metric [H]?

Thank you for your valuable comment. We understand the importance of evaluating calibration performance comprehensively. However, in our work, we follow the common practices established in the post-hoc calibration literature.

Specifically, Static Calibration Error (SCE) has been primarily used in training-time calibration scenarios, where issues like class imbalance during training are more prevalent. Since our work focuses on post-hoc calibration, which inherently does not involve addressing training-time class distribution imbalances, the motivation for using SCE is less clear in this context.

Instead, we used widely accepted metrics for post-hoc calibration, consistent with the literature, to ensure a fair evaluation of our methods against relevant baselines. We believe this approach maintains both alignment with existing research and comparability of results.

Q9. The intuition of using top-K softmax scores

We emphasize that the top-K class indices are first determined based on the softmax scores of the original input (not the transformed input). These class indices are then used to extract the corresponding softmax values from the transformed versions of the input (Section 3.3, and Algorithm 1 in the Appendix).

The intuition is that if the softmax values for these selected classes remain consistent across the transformed inputs and the original input, it suggests that the model's prediction for the original input is more likely to be correct.
This idea is inspired by the findings of Deng et al. (2022).

Q10. Why existing methods do not improve AUC compared to the proposed one

The reason is that our method explicitly learns to increase the confidence of correct predictions and reduce the confidence of incorrect ones, which is a key factor in improving AUROC (AUC). We have discussed this phenomenon in Section 4.3, lines 464–467. Other methods do not incorporate such a mechanism. Particularly in the case of narrowly incorrect predictions, other methods may even increase the confidence of incorrect predictions (Figure 3).

Q11. How good is the method in overcoming underconfidence of the model?

Our method effectively addresses both underconfidence and overconfidence issues. Please refer to the updated reliability diagram in the Appendix (D.1 RELIABILITY DIAGRAM) for further details.

Q1. Related work missing train-time calibration methods

Thank you for your valuable suggestion.
Our paper focuses specifically on post-hoc model calibration methods, as emphasized in the abstract and introduction. While train-time calibration (also referred to as pre-hoc calibration) is beyond the primary scope of our investigation, we acknowledge its relevance. We will incorporate a discussion on train-time calibration methods in the related work section to provide a more comprehensive overview of existing calibration approaches.

Q2. Reliability diagrams

We have added the reliability diagrams to the appendix (D.1 RELIABILITY DIAGRAM) in the revised paper. Kindly check our revised paper.

Q3. Why our method is able to improve OOD calibration performance? There is no related analysis

We have provided such an analysis. We kindly refer the reviewer to lines 452–454.

Our method is more effective on OOD test sets because they contain a higher proportion of narrowly incorrect predictions.

Further discussion on the impact of the number of narrowly incorrect predictions is also provided in Section 5 and Figure 5. The conclusion we have reached is that our method is superior when training and test sets contain more narrowly incorrect predictions.

Q4. How the proposed post-hoc calibrator would perform under class-imbalanced scenarios

1. Addressing class imbalance in the training set is typically not the focus in post-hoc calibration literature. This is because post-hoc calibration is conducted independently of the model's training process. It aims to adjust the output confidences of an already trained model without re-training or modifying model parameters, making it less sensitive to the specific data balance issues faced during model training.

2. While an extremely imbalanced calibration set could potentially negatively impact the calibration performance, there is no need to deliberately construct an imbalanced calibration set for training the calibrator. In practice, ensuring that the calibration set is reasonably representative of the testing distribution is generally sufficient to avoid any adverse effects from class imbalance.

Q5. What are the concrete differences between MDCA [C] loss and DCA loss [G]?

1. MDCA and DCA are auxiliary losses that should be used along with cross-entropy. Our analysis shows that cross-entropy has inherent issues in addressing narrowly incorrect samples in the post-hoc calibration problem. In contrast, our CA loss can be used independently.

2. Both MDCA [C] and DCA loss [G] aim to push the predicted confidence of the ground-truth class close to 1. However, they still face the "narrowly wrong prediction issue" in the post-hoc calibration problem, which is the core insight and motivation behind our CA loss.

Q6. Experimental results of replacing CA loss with MDCA loss and DCA loss

We keep the calibrator networks and inputs the same, replacing only the CA loss with the losses from MDCA [C] and DCA [G]. The results are shown below:

ImageNet-A:

ImageNet-R:

ImageNet-S:

ObjectNet:

We can observe that our CA loss outperforms DCA and MDCA in the post-hoc calibration task.

In reliability diagrams, comparisons should be made with relatively stronger post-hoc baselines, such as MIR or ProCal..

Esteemed Reviewer,

Thank you for your thoughtful feedback and questions, which have greatly contributed to improving our work.

In the latest revision, we have included the reliability diagrams for MIR and ProCal in Figure 7 and Figuire 8. We kindly invite you to review them.

Please do not hesitate to reach out with any further suggestions or questions.

Sincerely, The Authors

Q7. Evidence that your method is also capable of calibrating non-ground truth classes?

Theoretical evidence:

We kindly invite the reviewer to refer to our Gradient Computations and Case Analysis parts in our discussion with reviewer bjTY.

• When the prediction is correct, the CA loss not only calibrates the ground-truth class (with maximum softmax confidence) but also other classes, similar to the conventional cross-entropy loss.

• When the prediction is incorrect, scaling-based methods in the post-hoc system cannot reorder the logits' values, making it challenging to achieve the calibration goal for the ground-truth class. When using conventional cross-entropy loss to maximize the expectation on the ground-truth class, if the prediction is narrowly incorrect, the loss may make this prediction even sharper (Fig. 2). Therefore, to minimize the overall calibration error for such sample, we focus on calibrating the class with the maximum softmax probability (non-ground-truth classes). This non-ground-truth class can be more easily calibrated by learning a large temperature.

Experimental Evidence:

We evaluated the calibration performance of our method using metrics that consider the entire probability distribution:

• Expected Calibration Error (ECE): A lower ECE means that the predicted probabilities more accurately reflect the true likelihood of each class, not just the maximum.

• Brier Score: The Brier Score penalizes both overconfidence and underconfidence across all classes, so improvements (lower Brier score) here indicate better calibration of the entire probability vector.

Q8. SCE results

Thanks for your suggestion. The following table reports the Static Calibration Error (SCE) (%) on various datasets. The numbers are averaged from 10 models (the same as in the main paper).

We find that our method achieves competitive calibration performance when evaluated using the SCE metric.

Sincerely,

The Authors

We sincerely thank the reviewer for constructive feedback and positive comments that help us refine our work.

Q1. If these transformations do not adequately capture the characteristics of correct and incorrect predictions, the calibration might be less effective.

Thank you for raising this concern.

The core contribution of our method is to introduce a new calibration objective, which is derived from the definition of the calibration goal, and to analyze why it always aligns with the calibration goal, especially in narrowly incorrect predictions. Using transformations is an auxiliary technique that makes the learning objective easier to achieve. Even without transformations, our CA loss is still better than conventional maximum likelihood estimation (e.g., cross-entropy) methods. For example, in Tables 1, 2, and 3, "CA only" consistently shows an advantage compared to using cross-entropy loss.

Q2. Transfomations lack of theoretics

Generality of transformations:
In our method section and Fig. 2, we do not claim that our method is tied to specific transformations. Instead, we emphasize that using transformations can bring benefits, rather than emphasizing which transformations should be used. We also highlight that the core contribution of our method lies in the proposed CA loss, and our focus is on explaining why this loss function is superior to conventional MLE-based losses (e.g., cross-entropy loss and mean-squared error loss).

Selection of transformations:
Regarding the selection of transformations, we base our choices on findings in the literature [1,2,3]. Specifically, [1] demonstrated strong correlations between consistency under transformations (e.g., grayscale and rotation) and prediction correctness. Therefore, we empirically selected these two augmentations in our study. Additionally, we explored other commonly used augmentations [2] to demonstrate that our method is robust to various combinations of transformations.

Consistency across testing and training: During testing, the calibrator inputs are constructed using the same pipeline described in Fig. 2, ensuring that the same data augmentations are applied. This approach addresses potential concerns about discrepancies between training augmentations and the test image variance.

[1] Deng, W., Gould, S. and Zheng, L., 2022. On the strong correlation between model invariance and generalization. Advances in Neural Information Processing Systems, 35, pp.28052-28067.

[2] Zhong, Z., Zheng, L., Kang, G., Li, S. and Yang, Y., 2020, April. Random erasing data augmentation. In Proceedings of the AAAI conference on artificial intelligence (Vol. 34, No. 07, pp. 13001-13008).

[3] PyTorch Documentation (n.d.) Torchvision.transforms.ColorJitter. Available at: https://pytorch.org/vision/main/generated/torchvision.transforms.ColorJitter.html

Q3. What is the core difference between calibration and misclassification detection?

Calibration aims to align predicted probabilities with the true likelihood of correctness. It focuses on improving the reliability of the model’s probability estimates, not necessarily on identifying specific incorrect predictions. A calibrated model provides more trustworthy probability outputs, enabling better decision-making in probabilistic scenarios.

In contrast, misclassification detection aims to identify whether a specific prediction is likely to be incorrect. It focuses on the binary detection of correct vs. incorrect predictions, rather than refining the probability outputs. The outcome of misclassification detection highlights individual predictions likely to be errors, often used for error analysis.

We have cited the paper and added these insights into related work section.

Q4. The potential effects of even higher values of top-k, such as top 100 or top 200

Classifiers are typically trained to maximize the expectation of the ground truth class. As a result, the softmax output tends to assign significant confidence to only a few classes, while the rest are close to zero. In the ImageNet scenario, most classes have confidence values near zero. Therefore, if we increase "top-k" to 100 or even higher, most of the values in the top-k vector will still be close to zero, providing little additional useful information.

Q5. How the model manages scenarios where the correct prediction is not included among the top-4 predictions

Our method does not assume that the probability of the ground-truth class must be among the top k largest probabilities. Regardless of whether it is a correct or incorrect prediction, and irrespective of how large or small the probability of the ground-truth class is, it does not affect our ability to use the model's consistency on augmentations for the most confident (top-k) classes.

Q6. This method is more complex to implement when compared to simpler techniques because it uses a new loss function and transformed images.

(i) The CA loss is easy to implement and has an advantage over conventional MLE-based loss, as shown by our analysis of incorrect narrow predictions and experimental results.

(ii) The transformed image technique involves only a lightweight feed-forward network, where we select the top k probability values from each transformed image. The inference times for our method, temperature scaling, and PTS are similar: 2.33, 1.63, and 2.8 milliseconds per image, respectively. Even in a deployment with extremely limited computational resources, it is possible to use the CA loss alone, which is our core contribution.

Authors have addressed my concerns, I decide to keep my score.

We sincerely thank the reviewer for their constructive feedback and valuable comments, which help us improve the clarity and quality of our work.

Q1: The goal of the formal development (Section 3.2) is not clear

Thank you for your thoughtful feedback. We recognize that the goal of this section may not have been clearly articulated. Below, we clarify its objectives and address your concerns regarding its rigor and connection to the broader goal of demonstrating the validity of the empirical criterion.

Objective of the formal development. The primary goal of this section is to establish that the proposed empirical criterion $\mathbb{E}_f^{\text{emp}}$ is a sound and practical proxy for the theoretical calibration error $\mathbb{E}_f$. Specifically, it aims to: (i) Practical feasibility: Demonstrate how $\mathbb{E}_f^{\text{emp}}$ can be computed from observed data using the calibration pipeline depicted in Figure 2, which outputs only sample confidences $\hat{c}_i$. (ii) Theoretical approximation: Show that $\mathbb{E}_f^{\text{emp}}$ retains the key statistical properties of $\mathbb{E}_f$, despite being derived from the empirical distribution of observed data.

Connection between $\mathbb{E}_f^{\text{emp}}$ and $\mathbb{E}_f$ The derivation begins with the theoretical calibration error $\mathbb{E}_f$, which integrates over the true distribution $p(\hat{c})$. Since $p(\hat{c})$ is generally unavailable in practice, $\mathbb{E}_f^{\text{emp}}$ uses the empirical distribution derived from observed data. This substitution introduces a discretized approximation that preserves the essence of the theoretical formulation.

We demonstrate that this approximation simplifies the computation by using sample-level confidences and correctness indicators provided by the calibration pipeline. Furthermore, the analysis shows how the loss decomposes into terms representing confidence deviations, which directly relate to calibration goals.

Evaluation of $\mathbb{E}_f^{\text{emp}}$ as a proxy To validate $\mathbb{E}_f^{\text{emp}}$ as a meaningful proxy for $\mathbb{E}_f$, we derive its bounds and examine its behavior:

• Correct predictions: $\mathbb{E}_f^{\text{emp}}$ quantifies the deviation of predicted confidences from the ideal value of 1, reflecting underconfidence or overconfidence.
• Incorrect predictions: It penalizes overconfident predictions proportionally to their incorrectness, thereby addressing miscalibration directly.

These bounds confirm that $\mathbb{E}_f^{\text{emp}}$ effectively captures calibration quality, aligning with the theoretical intent of $\mathbb{E}_f$.

We acknowledge that $\mathbb{E}_f^{\text{emp}}$ introduces approximations due to the use of finite samples and discretization of the feature space. However, these limitations are inherent to empirical methods and are mitigated by the calibration pipeline's ability to generalize over observed data.

To further substantiate this, we have added:

1. Empirical simulations illustrating the convergence of $\mathbb{E}_f^{\text{emp}}$ to $\mathbb{E}_f$ with increasing dataset size.
2. A discussion of scenarios where the approximation may deviate significantly and its impact on the calibration pipeline’s performance.

We will revise the section to clearly articulate that its purpose is to establish $\mathbb{E}_f^{\text{emp}}$ as a computationally feasible and theoretically grounded approximation to $\mathbb{E}_f$. Additionally, we will explicitly:

• Clarify the assumptions and limitations introduced during the approximation process.
• Connect each step of the derivation to the outputs of the calibration pipeline in Figure 2.
• Highlight how the bounds for $\mathbb{E}_f^{\text{emp}}$ ensure its validity as a calibration measure.

Q2: The same symbol $E_{f}^{emp}$ for different concepts

$E_{f}^{emp}$ is used as the notation for the empirical calibration loss on function $f$ at different levels of discreteness. We will clarify this usage further in the revised version.

Q3: Why not directly predict the confidence instead of a temperature?

Using temperature ensures that the predicted class is not changed, thereby preserving the model's accuracy while only adjusting the sharpness or smoothness of the predictions. Our method effectively learns a temperature to smooth the predicted confidence vector, enabling correct predictions with a sharper confidence vector.

If we directly predict the confidence for the predicted class, it may alter the class index with the highest confidence, potentially changing the classification results.

Q4: What is the conceptual connection between the test-time data augmentation, the formal development, and the narrowly wrong sample analysis?

The formal development (from Equation 1 to our loss function, Equation 7) aims to explain that our loss is closely aligned with the definition of the calibration goal. Narrowly wrong sample analysis uses examples to demonstrate that, under certain conditions, conventional maximum likelihood estimation is not aligned with the calibration goal, whereas our learning objective is aligned with it.

However, directly learning this objective is not straightforward. In Section 3.3, we explained how test-time augmentation can help us achieve this learning objective.

Q5: What is the difference between CA only and CA trans?

"CA only" refers to using only the CA loss techniques to train the calibrator. "CA + trans" means using transformed images to prepare the calibrator input (as shown in Fig. 2). We will explain this part more clearly in the revision.

Q6: The approach focuses on calibration of the maximum confidence: can the strategy be adapted to calibrate the whole confidence vector (multiclass calibration)?

Thanks for raising this question; I would like to clarify how temperature scaling is used in our method.

Our approach involves learning a temperature parameter to perform sample-wise calibration of the prediction logits. The key point is that temperature scaling is applied uniformly to all logits, thereby scaling the entire confidence vector rather than just the maximum confidence. This means that our strategy inherently supports multiclass calibration by softening or sharpening the predicted probabilities for all classes simultaneously.

Q6. The temperature modifies the values of all scores. However it does not imply that the resulting scores for the other classes will be calibrated.

Part 1

Thank you for your valuable feedback and for bringing up an important point regarding the calibration of the entire multi-class probability vector. We provide a detailed explanation and clarify how our method impacts the calibration of all class probabilities, not just the maximum softmax score.

Gradient computations:

Our method employs sample-wise post-hoc temperature scaling, where we adjust the temperature parameter $ T $ for each sample to calibrate the model's confidence. The key idea is to make the maximum softmax confidence $ c $ close to 1 if the prediction is correct and close to 0 if the prediction is incorrect. Our loss function is defined as:

$

L = (c - \text{correctness})^2,
$

where $ c = \max_i p_i $ is the maximum softmax probability, and $\text{correctness}$ is 1 if the prediction is correct and 0 otherwise.

Since the temperature scaling affects all class probabilities due to the nature of the softmax function, optimizing $ T $ based on this loss function indirectly calibrates the probabilities of all classes. To demonstrate this, we provide a detailed gradient analysis.

The temperature-scaled softmax probabilities are given by:

$

p_i = \frac{\exp\left(\frac{z_i}{T}\right)}{\sum_{j} \exp\left(\frac{z_j}{T}\right)},
$

where $ z_i $ are the logits for each class, and $ T > 0 $ is the temperature parameter.

The gradient of the loss function $ L $ with respect to $ T $ is:

$

\frac{\partial L}{\partial T} = 2(c - \text{correctness}) \cdot \frac{\partial c}{\partial T}.
$

Thus, the key term to compute is $ \frac{\partial c}{\partial T} $, which depends on the gradient of the maximum softmax probability with respect to $ T $.

Computing $ \frac{\partial c}{\partial T} $:

Let $ k = \arg\max_i p_i $ be the index of the maximum softmax probability. Then:

$

c = p_k = \frac{\exp\left(\frac{z_k}{T}\right)}{\sum_{j} \exp\left(\frac{z_j}{T}\right)}.
$

The derivative of $ c $ with respect to $ T $ is:

$

\frac{\partial c}{\partial T} = \frac{\partial p_k}{\partial T} = p_k \left( \frac{\sum_{j} p_j \left( \frac{z_j}{T} \right) }{T} - \frac{z_k}{T^2} \right).
$

Simplifying, we get:

$

\frac{\partial c}{\partial T} = \frac{p_k}{T^2} \left( \sum_{j} p_j z_j - z_k \right).
$

Since $ \sum_{j} p_j z_j $ is the expected value of the logits under the probability distribution $ p_j $, the term $ \sum_{j} p_j z_j - z_k $ represents the difference between the expected logit and the maximum logit.

(See Part 2 for the rest of the content.)

Q6 Part2 (continued)

Case Analysis:

1. Correct predictions ($ \text{correctness} = 1 $):

   • Objective: Increase $ c $ towards 1.
   • Gradient direction: Since $ c < 1 $ and $ \text{correctness} = 1 $, the term $ (c - \text{correctness}) $ is negative.
   • Sign of $ \frac{\partial c}{\partial T} $: The term $ \sum_{j} p_j z_j - z_k $ is negative because $ z_k $ (the logit of the correct class) is higher than the expected logit $ \sum_{j} p_j z_j $. Therefore, $ \frac{\partial c}{\partial T} $ is negative.
   • Overall gradient: The product $ (c - \text{correctness}) \cdot \frac{\partial c}{\partial T} $ is positve (negative times negative), resulting in $ \frac{\partial L}{\partial T} > 0 $.
   • Effect on $ T $: A positive gradient $ \frac{\partial L}{\partial T} $ pushes $ T $ to decrease during optimization.
   • Impact on all class probabilities: Decreasing $ T $ sharpens the softmax distribution, increasing the confidence $ p_k $ in the correct class and decreasing the probabilities $ p_i $ of the other classes, meaning that the other classes are also calibrated. Therefore, we achieve a calibration effect on other classes similar to what Cross-Entropy provides.

2. Incorrect predictions ($ \text{correctness} = 0 $):

   • Objective: Decrease $ c $ towards 0.
   • Gradient direction: Since $ c > 0 $ and $ \text{correctness} = 0 $, the term $ (c - \text{correctness}) $ is positive.
   • Sign of $ \frac{\partial c}{\partial T} $: The term $ \sum_{j} p_j z_j - z_k $ is negative because $ z_k $ (the logit of the incorrectly predicted class) is higher than the expected logit. Thus, $ \frac{\partial c}{\partial T} $ is negative.
   • Overall gradient: The product $ (c - \text{correctness}) \cdot \frac{\partial c}{\partial T} $ is negative (positive times negative), resulting in $ \frac{\partial L}{\partial T} < 0 $.
   • Effect on $ T $: A negative gradient $ \frac{\partial L}{\partial T} $ pushes $ T $ to increase during optimization.
   • Impact on all class probabilities: Increasing $T$ smoothens the softmax distribution, reducing the confidence $p_k$ in the incorrectly predicted class. For other classes, the ideal outcome is to increase the confidence of the ground-truth class while decreasing the remaining ones. However, it is important to be clear about the constraints of scaling-based post-hoc calibration: it cannot change the order of the logits, meaning it cannot perfectly calibrate each class as expected by cross-entropy loss. It can only make the softmax vector sharper or smoother. As a result, it is challenging to increase the confidence value of the ground-truth class due to the limitations of the softmax operation. More importantly, cross-entropy may result in an undesired temperature, potentially making the incorrect prediction more confident (as shown in Fig. 2), even though cross-entropy seemingly calibrates across all classes directly. In contrast, our method focuses on reducing the maximum confidence value, which is a practical approach to significantly reduce the calibration error, particularly in incorrectly predicted samples.

Experimental Evidence:

We evaluated the calibration performance of our method using metrics that consider the entire probability distribution:

• Expected Calibration Error (ECE): A lower ECE means that the predicted probabilities more accurately reflect the true likelihood of each class, not just the maximum.

• Brier Score: The Brier Score penalizes both overconfidence and underconfidence across all classes, so improvements here indicate better calibration of the entire probability vector.

Thank you again for your insightful comments. Feel free to reach out with any additional suggestions or questions.

Sincerely,

The Authors

Q1. Formal Development

$ p(x \mid c) $ is not well-defined.
Thank you for your comment. $ p(x \mid c) $ represents the probability distribution of $ x $, where the confidence of $ x $ is equal to $ c $.

Equation 5 is not a true empirical criterion.
To obtain the empirical calibration loss, we gradually approximate Equation (3) through empirical estimation. Equation (5) is an intermediate step in the discretization process. We will revise this part to make it more clear.

In Equation 6, the summation used for estimating the conditional accuracy is switched before the norm.
Thank you for pointing out the issue in the transition from Eq. (5) to Eq. (6). We acknowledge that the transition from Eq. (5) to Eq. (6) is not a strict mathematical equivalence but rather an empirical approximation. We have revised it to make the explanation more rigorous.

On the other hand, whether the summation is inside or outside the norm, both are approximations of the calibration loss, and the development from Eq. (6) to Eq. (7) remains the same. Therefore, the loss function we use in Eq. (7) is not affected by this adjustment.

End up with a sample-based criterion, but which is precisely the Brier score.
We respectfully disagree with this statement. The Brier score is the discrepancy between the predicted softmax confidence vector and the one-hot ground truth vector. In contrast, our CA loss computes the discrepancy between the confidence value and the correctness (0 or 1), which is fundamentally different in its definition and purpose.

Q3. Why do you need to express this correction as a modified softmax temperature since the probabilities of the other classes are not involved in the calibration objective?

I'd like to clarify why expressing the correction as a modified softmax temperature is essential, even though our calibration objective focuses on the maximum softmax score.

1. The softmax temperature influences the maximum score directly

The softmax function computes probabilities based on the relative differences between the logits of all classes. By adjusting the temperature parameter, we scale these differences:

• Higher Temperature: Softens the probability distribution, making the probabilities more uniform and reducing the maximum softmax score.
• Lower Temperature: Sharpens the distribution, increasing the maximum softmax score.

This scaling directly affects the maximum softmax score, allowing us to calibrate it toward 1 when the prediction is correct and toward 0 when it's incorrect.

2. Maintaining a valid probability distribution

Adjusting the temperature modifies all class probabilities while ensuring they still sum to 1. This is crucial because:

• Probabilistic Integrity: We maintain a valid probability distribution, which is important for interpretability and further probabilistic reasoning.
• Consistency Across Predictions: It ensures that the calibration does not produce anomalous probability distributions that could negatively affect downstream tasks or metrics.

3. Smooth and principled adjustment mechanism

Using the temperature parameter allows for a smooth and continuous adjustment of the softmax outputs:

• Avoiding Abrupt Changes: Directly altering the maximum score without considering other classes could lead to abrupt or unprincipled changes in the probability distribution.
• Theoretical Foundation: Temperature scaling is a well-established method in calibration literature (e.g., Guo et al., 2017), providing a theoretically sound approach to adjust model confidence.

I hope this explanation clarifies the necessity of expressing the correction as a modified softmax temperature.

• Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017). On Calibration of Modern Neural Networks. Proceedings of the 34th International Conference on Machine Learning, PMLR 70:1321-1330.

Q4. Why the empirical loss solves the calibration problem.

The CA loss in Eq. (7) is an approximation of the integral (Eq. (3)) of the density function (Eq. (2)), where Eq. (2) is based on the definition of model calibration (Eq. (1)). Therefore, minimizing the CA loss is a step towards achieving model calibration.

Q3 & Q6 (1) Authors use the binary ECE metric, which is standard in the field, rather than the class-wise version, so using a global temperature is not necessary..

(1) The classical post-hoc model calibration method, Temperature Scaling [1], which is widely acknowledged in the literature (with 6000+ citations), uses a temperature to "globally" scale the entire logits vector (across all classes). The evaluation metric used in their work is also the standard Expected Calibration Error (ECE), which is the same metric that we use. Therefore, we do not believe that our evaluation setup is problematic and we also do not think using the temperature to scale all classes is problematic.

(2) Regarding the class-wise version of the calibration error, we understand that the reviewer might be referring to Static Calibration Error (SCE), which evaluates the calibration error for each class individually. While this metric is not commonly used in existing post-hoc model calibration literature, we have still included its results in our discussion with Reviewer LQKt. Please refer to our discussion with Reviewer LQKt for details on the evaluation results using the class-wise calibration error.

[1] Guo, C., Pleiss, G., Sun, Y. and Weinberger, K.Q., 2017, July. On calibration of modern neural networks. In International conference on machine learning (pp. 1321-1330). PMLR.

Q3 & Q6 (2) Calculations give some insight but not show that the CA loss solves the calibration problem when the number of samples goes to infinity.

We thank you for indicating that our previous discussion provided some insight. Regarding this question, as the sample size approaches infinity, we can access the distribution of $p(x \mid \hat{c})$, allowing us to directly use Equation (2) to compute the theoretical calibration error.

Thank you again for your comments.

Sincerely,

The Authors

Q7. Parameter $\theta$ in Equation 19

In Eq. 19, $\theta$ represents the calibrator weights, as noted in Line 298 of the main paper.

Q8. What is the synergy between CA Loss and Transformation Component? Applying CA loss reduces ECE, while the transformation tends to increase ECE, showing a trade-off rather than synergy.

Synergy between CA Loss and transformation component:
CA Loss helps the calibrator learn to adjust the confidence of both correct and incorrect predictions. This requires the post-hoc calibrator to be aware of the correctness of each prediction. Existing work (Deng et al., 2022) has shown that consistency in model predictions for transformed images effectively indicates prediction correctness. CA Loss provides the learning objective, while augmentation techniques make achieving this objective easier.

Transformation tends to increase ECE, showing a trade-off rather than synergy:

We believe there is another way to interpret this.

In Tables 1, 2, and 3, combining CA Loss with confidence values from transformations results in lower ECE than using CA Loss alone in 8 out of 10 datasets, demonstrating that the two components work synergistically rather than exhibiting a trade-off.

Q9. Results on alternative baselines

Our transformation-based calibrator can unlock the potential of CA loss. Meanwhile, our CA loss also provides positive improvements to other calibrators, as demonstrated in Table 1, 2, and 3, where it enhances PTS.

We also replace the cross-entropy (CE) loss with correctness-aware (CA) loss in the temperature scaling (TS) method. Below, we present our experimantal results on ImageNet-A, ImageNet-R, ImageNet-S and ObjectNet.

ImageNet-A:

ImageNet-R:

ImageNet-S:

ObjectNet:

Results show that our method also provides improvements. However, the results here are not as significant as those achieved with CA loss combined with our calibrator (with image transformations). This is because the consistency across image transformations provides informative features that help achieve the learning objective of CA loss.

Q10. Minor points

Thanks for your suggestion. We have addressed them in our revised paper.

Q11. Why the paper emphasizes using a continuous calibration error instead of the Expected Calibration Error (ECE)?

Kindly refer to our response in Q4.

Q12. What is the intended synergy between the CA loss and the transformation component?

Kindly refer to our response in Q8.

Q13. Results on alternative baselines

Kindly refer to our response in Q9.

Q1. A clear definition of "narrow misclassification" is missing. How it differs from absolutely wrong samples.

Thank you for raising your confusion.
We have defined "narrowly wrong sample/prediction" (which has the same meaning as "narrow misclassification") in lines 348 and 377. Additionally, we provided the definition of "absolutely wrong samples" in line 342.

The examples in Fig. 3 also offer a comprehensive explanation of why "narrowly wrong predictions" can have a negative impact during learning when using traditional Maximum Likelihood Estimation methods (e.g., cross-entropy loss).

Q2. Lack of analysis of how different types of augmentation affect performance across domains.

1. Our method does not impose constraints on the selection of augmentations, as illustrated in Figure 2. The optimization of augmentation combinations is beyond the scope of this paper.

2. Considering the vast number of potential datasets in the real world, determining the "best" augmentation combination would be impractical and of limited value. Instead, our experiments are designed to demonstrate that using multiple augmentations consistently yields better results than using a single augmentation (Figure 4), particularly in terms of improving calibration with our CA loss. However, augmentations beyond three do not necessarily lead to better performance.

Q3. The uniqueness of the proposed algorithm’s approach in addressing "narrow misclassification."

The reviewer may not have fully appreciated the challenge of narrowly incorrect predictions in Maximum Likelihood Estimation (MLE) or the detailed workings of our proposed method.

• MLE always aims to maximize the confidence in the ground truth class. For a "narrowly wrong prediction" (also known as "narrow misclassification"), if we use a temperature to adjust the logits of all classes to achieve higher confidence in the ground truth class, the converged temperature may be less than 1 (as shown in Fig. 3). This can result in the prediction confidence (i.e., the confidence in the wrongly predicted class) for this narrowly wrong sample becoming higher after calibration.

• In contrast, for any wrong prediction—whether it is a narrowly wrong prediction or an absolutely wrong prediction—our method ensures that the temperature converges to a value greater than 1 on expectation. By directly adjusting the logit value of the ground truth class, our approach aims to reduce the confidence of wrong predictions, aligning with the objective derived from the definition of calibration.

Q4. Why continuous assumptions were initially made if the final derivation closely resembles an ECE-based approach.

The conventional bin-based ECE is widely considered an important metric for evaluating calibration, but technically it cannot be used as a training loss (non-differentiability). Therefore, existing methods usually use conventional cross-entropy loss (e.g., TS (Guo et al., 2017), PTS (Tomani et al., 2022), Adaptive TS (Joy et al., 2023)), which we have discussed has issues with narrowly wrong predictions.

• Our method begins by using the formal definition of model calibration and derives the expected calibration error in a distributional sense. From there, we gradually discretize the calibration error to formulate our Equation 7.

• While Equation 7 serves as an approximation of the expectation of calibration error, it can be directly optimized for calibrator training. Although it appears similar to a continuous ECE formulation, its derivation is fundamentally different, originating from a distributional perspective rather than the traditional ECE evaluation metric.

Q5. The significance of the bounds of CA loss remains unclear

The significance of the bounds for the CA loss lies in understanding its behavior during training:

• Lower bound: Demonstrates whether the loss function can converge during the optimization process, ensuring stability and effectiveness.

• Upper bound: Indicates the maximum possible value of the loss, useful for diagnosing training stability and ensuring expected behavior.

Establishing both bounds validates the reliability and applicability of the proposed loss function.

Q6. The derivation in Equation 15 is ambiguous due to an unexplained arrow.

The expression $\frac{1-\rho}{C}$ denotes the lower bound of $\mathbb{E}_f^\text{emp}$, as shown in Eq. (11). The arrow ($\rightarrow$) indicates the optimization of $\mathbb{E}_f^\text{emp}$ towards this lower bound, $\frac{1-\rho}{C}$, which is equivalent to minimizing $\mathbb{E}^\text{diff}$ towards the lower bound $\frac{\Big(\frac{1}{C} - 1\Big)\rho}{1-\rho}$. This process involves pushing the average maximum softmax scores of incorrectly classified samples away from those of correctly classified samples, thereby reducing the overlap of confidence values between correct and incorrect predictions (Lines 250-254 of our main paper).

The meanings of the previously confused terms have been clarified after the rebuttal. I hope the authors will revise the paper to better explain the definitions provided in the comments. Specifically, I expect the additional measurements to be included in the appendix or elsewhere. The additional experiments impressively demonstrate the generalizability of the proposed algorithm. Believing that these points will be addressed, I have decided to increase my score.