Beyond One-Hot Labels: Semantic Mixing for Model Calibration

Response to Reviewer jJxJ

Thanks for your helpful suggestions! Here’s our response:

Q4-1: Training computational cost and memory usage compared to existing methods.

A4-1: As also analyzed in A2-3, we compare the computational efficiency in terms of the training time in A2-3 Table B:

We can conclude that the number of augmented samples per batch is the major factor in training time. Our CSM maintain relative efficiency considering the calibration effectiveness. When compared on the EQ-DATA setting in the main paper Table 2, CSM achieves competitive calibration results with an equalized training time.

Such factor also decides memory usage. CSM uses ~$\frac{3}{2}$ memory compared to Mixup/RegMixup, while using ~equal memory compared to the RankMixup (depending on setep).

Regarding resource consumption, we generate augmented samples and train our model on one A4000 device. It's worth noting that CSM needs no re-generation when switching model/objectives/re-annotation methods, making it more efficient for decoupled study of these modules.

We will include the key information in the main paper.

Q4-2: Hyperparameter sensitivity.

A4-2: We have analyzed hyperparameter $s$ and the number of augmented samples in Appendix C. We provide more analyses about the number of augmented samples per training sample (denoted as $N_{aug}$) here. Table C:

It can be observed that adding the number of accompanied augmentations per dataset sample can generally improve the final calibration performance. This is because a larger number of $N_{augs}$ can sample more sufficient proximal data for training, better filling the domain space and providing more accurate confidence estimation.

For sensitivity analysis, we have the following key observations to enhance our experiments considering existing results in Appendix C:

1. Our proposed CSM is not sensitive to the total number of augmented samples. A relatively smaller quantity can still make CSM effective.
2. CSM is relatively sensitive to the scaling factor $s$ as it is related to the temperature. Nevertheless, under an appropriate range of $s$, the method can perform consistently well.

Q4-3: Evaluation on the transformer architecture.

A4-3: Evaluations on the Swin-Transformer architecture verify our method's equal or stronger effectiveness. Please refer to A1-3 Table A.

Q4-4: Failure cases of CSM.

A4-4: While our method can perform well on all the evaluated datasets, we admit there are some cases CSM fails. For instance, although CSM surpass compared methods on ECE/AECE, some post-temperature results on the CIFAR-100 dataset are not comparative to the SOTA methods. Specifically, they exhibit balanced pre- and post-temperature results (searched $T = 1$) with remarkable pre-temperature ECE values but slightly larger calibration errors after temperature scaling. Such phenomenon is also stated in the calibration literature [1], where they found a balance exists in the two results. Considering both results together, our CSM can still achieve satisfactory model calibration.

Q4-5: Will the choice of generative models affect the performance?

A4-5: The choice of the generative model does influence the final prediction results. Some existing study [2] has already proved that adopting a better generative model could improve the classification model's robustness. We anticipate that a better generative backbone can achieve superior confidence calibration using our CSM. This aligns with our empirical experiences that using the ordinary Stable Diffusion architecture, we can only achieve suboptimal results as shown in the table:

Therefore, we anticipate that a typical GAN with less parameters or lower fidelity compared to diffusion models would yield worse results.

References

[1] Wang, D. B., Feng, L., & Zhang, M. L. (2021). Rethinking calibration of deep neural networks: Do not be afraid of overconfidence. Advances in Neural Information Processing Systems, 34, 11809-11820.

[2] Wang, Z., Pang, T., Du, C., Lin, M., Liu, W., & Yan, S. (2023, July). Better diffusion models further improve adversarial training. In International conference on machine learning (pp. 36246-36263). PMLR.

[3] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022). High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 10684-10695).

[4] Karras, T., Aittala, M., Aila, T., & Laine, S. (2022). Elucidating the design space of diffusion-based generative models. Advances in neural information processing systems, 35, 26565-26577.

Response to Reviewer WPRF

Thank you for your encouraging feedback on the clarity, soundness, and comprehensive evaluation of our work. We truly appreciate your thoughtful suggestions for clarity and comprehensive validation. Here are our responses to the suggestions:

Q3-1: Clarify Proposition 3.4 (L2 loss vs. CE/FL).

A3-1: Thank you for commenting on the clarity issue. We need to clarify that there exists a typo in Proposition 3.4 potentially hindered understanding. Proposition 3.4 should have been presented as

• $\forall \delta \ge 0$, we have $\beta(p^{L2}_1, p^{L2}_2) = 0$,

which is an equation rather than an inequality for the $\mathcal{L}_2$ loss's balance function, meaning that when two similar samples exceed the model's discriminability, $\mathcal{L}_2$ loss tends to balance the learned labels of the harder and softer instances, instead of tending to fit a specific one of them. Note that the proof of Proposition 3.4 we provided in Appendix A.3 does prove that $\beta(p^{L2}_1,p^{L2}_2) = 0$.

As proved in Appendix A, easier samples with $\delta \ge \|q^{L2}_1, q^{L2}_2\|$ would generally have $\|p^{L2}_i, q^{L2}_i\| = 0, i=1,2$. Smaller $\delta$s indicate difficulty for the learned model to separate the outputs for both samples, hence introducing a balancing problem. A theoretically non-zero balancing score $\beta$ means one of $\|p^{L2}_i, q^{L2}_i\|, i=1,2$, is minimized more completely, indicating the imbalanced nature of the objective.

Among the three, only the L2 loss theoretically zeros the $\beta$ balancing score, indicating its ability to balance over- and under-confidence of difficult soft-labeled sample pairs, hence being a superior loss for calibration with our augmented samples. In practice, the soft-label distribution may disturb such balance, which can be one of our future study.

We will correct this typo in the revised main paper.

Q3-2: Baseline for directly adopting CLIP labels. (Var. 1)

Q3-3: Baseline for diffusive sample augment without semantic mixing. (Var. 2)

A3-2, 3-3: We evaluate these two variants and compare them with our proposed CSM as follows: CIFAR-10:

CIFAR-100:

From these results, we can have these valuable observations:

1. The vanilla CLIP annotation method yields the worst ACC and pre-temperature calibration errors, primarily due to the noisy information by annotating all classes. Such degradation is significant for CIFAR-100, in which there are more classes so that the noise is severer.
2. Directly adopting class-conditioned augmentations from the diffusion model can slightly rise the prediction accuracy, as also evidenced by the generative model augmented classification literature. However, as it does not contain soft-labeled samples, Var. 2 fails to improve model calibration.

Therefore, we can conclude that models are effectively calibrated only when adopting the proper data and re-annotation scheme.

Response to Reviewer xs6o

Thank you for your positive and insightful feedback! Here are our responses:

Q2-1: Additional details and proofs in the supplementary material.

A2-1: Thank you for the kindly comments on the theoretical soundness. The claims made in our paper (including deducted results in Eq.(6)-(10) and Prop. 3.2-3.4) are sufficiently proved in Appendix A. In detail:

• Eq.(6)-(7) corresponds to Eq.(14)-(15) with
  • Assumption 1: Classification optimal classifier $\operatorname{E}(\cdot)$ ensures the likelihood ratio of different classes;
  • Assumption 2: Regarding the feature as the affine set elements of class prototypes with an orthogonal deviation.
• Eq.(8)-(9) is proved by Eq.(15)-(19).
• The result of Eq.(10) is acquired from Eq.(9) with the class factor invariance assumption.
• Proposition 3.4 is first proved through Eq.(22)-(26) with the assumptions given in Definition 3.1.
• To prove Proposition 3.2 and 3.3, we first give a general analysis of the problem in Line 712-791 (or Eq.(27)-(37)), then prove them by Lemma A.1 and Lemma A.2, respectively. Note that the proof for CE is unconditional, while for FL it's proved with assumption that $\gamma_{FL}=1.0$ and we empirically find FL more imbalanced with larger $\gamma_{FL}$.

These detailed descriptions illustrate the overall framework of the theoretical analysis. We will include connective details and key deductions in the main paper.

Q2-2: Additional evaluations on other architectures.

A2-2: Our method performs equally or more effectively to others with the Swin-Transformer architecture. Please refer to A1-3 Table A.

Q2-3: More details on efficiency and resource consumption.

A2-3: Thank your for your suggestion. For efficiency analysis, we compare explicitly in terms of the training time as follows:

Table B:

One can see the number of augmented samples per batch is the major factor for training time. CSM outperforms others in ECE/AECE while maintaining reasonable speed. Even with equalized training samples (EQ-DATA, Table 2), it achieves competitive calibration. CSM runs on a single A4000. Augmented samples need no re-generation across model/loss/annotation changes, enabling efficient modular study.

Key details will be added to the main paper.

Q2-4: Comparison with [1], a test-time augmentation (TTA) post-hoc calibration method.

A2-4: We compare with [1] by evaluating CSM + [1] as follows:

Our integrated method balances ECE and AECE, achieving an optimized AECE of 1.53 on CIFAR-100. Compared to [1] using test-time sample-wise scaling, CSM employs training-time augmentation with inter-sample augmentations to expand the proximity space, enhancing calibration robustness. We will cite [1] and provide full comparisons in the main paper.

Q2-5: Computational overhead of diffusion-based augmentation.

A2-5: Our analysis in Appendix C shows that CSM requires few augmented samples to launch effectively, ensuring low computational costs. Generating augmented sets takes ~4 hours (CIFAR-10/100) or ~9 hours (Tiny-ImageNet) on an RTX4090 GPU, comparable to typical training times. Crucially, CSM eliminates re-generation when model architectures/parameters change, further enhancing efficiency through its decoupled design. This validates CSM's computational efficiency.

Q2-6: Hyperparameter sensitivity analysis.

A2-6: We have analyzed parameter $s$ and No. of augmented samples in Appendix C. We analyze $N_{aug}$ (refer to A4-2 Table C) and check sensitivity in A4-2. Larger $N_{aug}$ generally enhances our method, while within appropriate ranges of other parameters, it yields stable results.

Q2-7 Possibility to integrate our CSM with [2] to enhance calibration and robustness.

A2-7: Thank you for this insightful question. We conduct post-hoc experiments to integrate our method's outputs with [2], acquiring the following result:

Due to limited time, we simply integrate CSM with [2] without further adjustments. Although a simple combination of them doesn't yield superioir ECE/AECE results, we find that the proximity-informed metric PIECE displays better results, which validates the robustness growth related to proximity from the integration. We will cite [2] for analysis.

References

[1] Hekler, A., Brinker, T. J., & Buettner, F. (2023, June). Test time augmentation meets post-hoc calibration: uncertainty quantification under real-world conditions. AAAI.

[2] Xiong, M., Deng, A., Koh, P. W. W., Wu, J., Li, S., Xu, J., & Hooi, B. (2023). Proximity-informed calibration for deep neural networks. NeurIPS.

Response to Reviewer TsGd

Thank you for your kind suggestions on clarity and experimental thoroughness. Below are our responses:

Q1-1: Clarification on the reason that the proposed L2 loss is a balanced loss.

A1-2: Thank you for this nice concern. We need to clarify that there exists a typo in Proposition 3.4 which makes the conclusion confusing. Proposition 3.4 should have been presented as

• $\forall \delta \ge 0$, we have $\beta(p^{L2}_1,p^{L2}_2)=0$,

which is an equation rather than an inequality for the $\mathcal{L}_2$ loss's balance function, meaning that when two similar samples exceed the model's discriminability, $\mathcal{L}_2$ loss tends to balance the learned labels of the harder and softer instances, instead of tending to fit a specific one of them.

Note that the proof of Proposition 3.4 we provided in the Appendix A.3 does prove that $\beta(p^{L2}_1,p^{L2}_2) = 0$. With such theoretical justification, we also present an empirical evidence in A1-5 regarding the confidence balance score.

Also refer to A3-1. We will correct this typo in the revised main paper.

Q1-2: Missing comparisons with ACLS [1] and CALS [2].

A1-2: We compare our result with theirs in the following tables. The results demonstrate the competitive or superior performance of our method compared to the state-of-the-arts. We will cite these compared methods and include these results in the final paper.

Q1-3: Insufficient validation on ImageNet and Transformers.

A1-3: We compare our result with representative methods on ImageNet with the ResNet-50 and Swin-Transformer architectures. Our method performs equally or more effectively compared to these methods, especially to the mixup-based methods. We will include these results in the final paper.

Table A:

Q1-4: Potential reversal of the following CE/FL explanations in ablation (lines 382–384).

"In contrast, CE and FL losses often require temperature adjustments, with CE favoring sharper labels and FL for softer ones, aligning with our theoretical expectations from Section 3.3."

A1-4: The analysis corresponds to the searched temperature values of Mixup and CSM in Table 4, where CE results sometimes involve a searched temperature larger than 1.0 compared using Mixup (Mixup (CE): T = 1.3), while FL results produce a searched temperature of T = 0.9 < 1 compared with our CSM (CSM (FL)). These two specific results highlight the nature of CE and FL losses.

As studied by existing works, a higher post-temperature can indicate the model's over-confidence while a lower one suggests under-confidence of the pre-temperature model. Therefore, with soft labels during training, these phenomena indicate a preference/bias of fitting different samples for the adopted losses, i.e., harder labels (e.g., close to one hot) vs. softer labels (e.g., mixup labels with $\lambda=0.5$).

Q1-5: Confidence-balancing comparison across CE, FL, and L2.

A1-5: We explicitly compute the average balance scores to illustrate the confidence balancing results here:

Our loss shows a clear confidence balance between CE and FL, confirming its effectiveness, though empirical values are typically negative due to the rareness of indistinguishable pairs and easier learning of high-confidence samples in practical experiments.

References

[1] Park, H., Noh, J., Oh, Y., Baek, D., & Ham, B. (2023). Acls: Adaptive and conditional label smoothing for network calibration. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 3936-3945).

[2] Liu, B., Rony, J., Galdran, A., Dolz, J., & Ben Ayed, I. (2023). Class adaptive network calibration. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 16070-16079).