Multiclass Alignment of Confidences and Softened Target Occurrences for Train-time Calibration

1. The innovation is weak. The method, (Targets softening) is very similar to PSKD [Self-Knowledge Distillation with Progressive Refinement of Targets ]
2. The evaluated architecture is limited, only resent involved.

• The technical contribution of the paper is limited. The contribution stated in the paper has been published earlier [1] in some way, but no citation is provided for the publication.
• The manuscript has a problem with plagiarism (please see the Ethics concerns for more details).
• This paper mainly focused on the classification task, whereas the previously published paper was based on object detection.

The proposed work does bear novelty: combining knowledge distillation for label smoothing and replacing NLL/FL with Pearson correlation for smoothed results. However, these contributions may not meet the high standard of ICLR, as both self-distillation and smoothness-oriented losses (e.g., FL) have been widely discussed in uncertainty estimation / calibration, while the authors have not made significant theoretical breakthroughs nor significantly better empirical results in the current shape of submission.

There is unfortunately a lack of clarity in the very core arguments: Sec 3.2: “The loss formulation is inspired by the intuition that as training goes, a model’s prediction becomes refined, and thus the predicted confidence scores can be gradually combined with the ground truth, to form a smoothed target distribution which has an increased entropy compared to the one-hot encoded hard targets, potentially leading to a better calibrated model.” This should be the most important argument supporting the manuscript, but what does this mean? Can the authors please make breaks to improve the clarity?

After Eq. 3: “Additionally, in multiclass calibration, we care about preserving the class relations. Pearson correlation-based loss function allows the model to be guided appropriately to distill those truly informative multiclass relations.” What does “class relation” here mean? The argument of this paragraph is not intuitive. Instead, for multi-class scenarios, KL divergence is also computed for multiple classes as well.

Adding smoothness constraints often comes at the cost of losing sharpness, therefore hurting categorical accuracy: the authors are therefore encouraged to report acc for the upper part of Table 2. Even looking at the lower half of Table 2, the proposed approach sometimes loses sharpness compared with NLL + MbLS. Also, what does NLL/FL in the lower part of Table 2 mean? Are they from NLL or FL or from a linear combination of them?

Despite special considerations for multi-class tasks, when checking SCE (measuring multi-class calibration) and Acc, it is difficult to see if the proposed approach yields better multi-class results than NLL + MbLS

Given that knowledge distillation, including this work, has been widely applied for label smoothing, the authors are encouraged to discuss at least the following similar works, and make comparisons if they find it necessary: [2-4]

[1] ACLS: Adaptive and Conditional Label Smoothing for Network Calibration [2] Self-Distribution Distillation: Efficient Uncertainty Estimation [3] Efficient Uncertainty Estimation in Semantic Segmentation via Distillation [4] Distilling Calibrated Student from an Uncalibrated Teacher

1. The introduction contents and design overlap a lot with the cited MDCA work [1].
2. The experiments show marginal gains from the proposed gradual target softening. The benefits of this key component are not strongly demonstrated, weakening the overall contribution.
3. The overall performance is similar to NLL/FL+MDCA, while introducing an additional hyperparameter for the gradual softening. The improvements over existing methods are incremental, and not clearly significant. The authors should provide statistical significance testing and quantify the differences to prior work. Small incremental gains may not be adequately justified as a stand-alone contribution.

[1] Ramya Hebbalaguppe, Jatin Prakash, Neelabh Madan, and Chetan Arora. A stitch in time saves nine: A train-time regularizing loss for improved neural network calibration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16081–16090, 2022.