Tailoring Mixup to Data for Calibration

We thank the reviewer for their review. We answer to their questions and remarks below.

• W1: In the revision, we introduce a new corollary derived from Theorem 5.1 in Liu et al. [A], showing that the probability of assigning a wrong label with mixup is lower bounded by the distance between the manifolds of the original data used for mixing. We refer the reviewer to our answer W1 to reviewer XhfP for the full derivation and discussion and Theorem 3.2 in the revision. This novel theoretical result, which is confirmed by the experiments in Figure 1a, better supports and motivates our method.
• W2: We compare our method to recently published Mixup methods designed to improve calibration on a wide range of benchmarks.
• Q1: This is a good suggestion. By defining fixed Beta distributions depending on the distance, we can have a more efficient (in terms of computation time) implementation. However, we also expect worse results as the method lacks granularity in its behavior. We present below experiments with a ResNet-101 on CIFAR 10 and CIFAR 100, with the same settings as results in Table 2. We defined two separate Beta distributions, one with $\alpha_{low}$ parameter, for pairs with distance lower than average, and the other with $\alpha_{high}$ parameter, for pairs with distance higher than average. We will add this comparison and discussion in the Appendix.

• Q2 + Q3: Combining our method (or other calibration-driven method) with non-linear mixup for images, like CutMix [B], can be done, since the changes are in different parts to each other. However, extensive experiments in Pinto et al. [C] (in their Appendix H) have shown that combining their method with CutMix is not trivial as it does not always improve results and robustness. We also expect that introducing a non-linear mixing operation will not lead to the same improvements of calibration with our method, since theoretical groundings and intuitions are always based on the linear mixing operation. Recent results [D] has also shown that the distortion of the training loss induced by non-linear mixup methods, like CutMix, leads to different and less optimal decision boundaries. We leave for future work the theoretical derivations of using the non-linear mixing operation with the aim to improve calibration and robustness.
  Nonetheless, we present below first results with CutMix and SK CutMix, in which we apply our similarity kernel to change dynamically the Beta distribution in CutMix, in the same settings as Table 2. We can see that SK CutMix can improve accuracy over CutMix. Although SK CutMix can also improve calibration over CutMix, since CutMix does not improve calibration over Mixup, SK CutMix does not lead to better calibration than SK Mixup. We will add this discussion and a full comparison in the revision.

• Q4: In the reference, methods are evaluated either with pretrained checkpoints from internet, or with ResNet-50 trained with 300 epochs. In our experiments on ImageNet, we trained all methods for 100 epochs on Imagenet, following similar settings as Pinto et al. [C] (detailed training settings in our Appendix H).
  We also trained a ViT for 300 epochs on Imagenet (detailed training settings in our Appendix H), and we obtain better results on ImageNet A. We show the complete benchmark with ViT in the table below.

  Methods	IN Acc	IN ECE	IN AECE	IN-R Acc	IN-R ECE	IN-R AECE	IN-A Acc	IN-A ECE	IN-A AECE
  ERM	69.34	9.03	9.03	15.46	35.93	35.93	1.83	45.03	45.03
  Mixup	72.0	5.81	5.8	18.21	29.29	29.29	2.47	41.07	41.07
  Manifold Mixup	72.04	6.99	6.95	18.93	30.27	30.27	2.15	43.47	43.47
  RegMixup	74.44	7.36	7.34	21.01	32.64	32.64	3.8	44.4	44.4
  RankMixup	69.99	9.3	9.3	15.77	37.37	37.37	1.77	46.82	46.82
  MIT-A	72.81	5.69	5.69	17.6	31.07	31.07	2.81	41.8	41.8
  SK-Mixup (Ours)	71.83	3.89	3.9	18.37	28.56	28.56	2.28	39.33	39.33
  SK RegMixup (Ours)	75.11	2.0	1.98	22.06	26.23	26.23	4.44	38.25	38.25

We thank the reviewer for their review.

• W1: This is a very insightful remark. We can obtain this result by introducing a new corollary from Theorem 5.1 in Liu et al. [A]. Let's consider $X, X'$ random variables associated to the inputs, and $\tilde{X} = (1 - \lambda) X + \lambda X'$, for a fixed $\lambda \in [0,1]$. The ground truth conditional distribution of the label $Y$ is expressed as a vector-valued function $f: \mathbb{X} \rightarrow \mathbb{R}^M$, such that $P(Y = m | X = x) = f_m(x)$, for each dimension $m \in \{1, \dots, M\}$. The mixup-induced label can then be assigned based on ground truth $\tilde{Y}^* = arg max_m f_m(\tilde{X})$, or from the combination of conditional distribution $\tilde{Y} = arg max_m [(1 - \lambda) f_m(X) + \lambda f_m(X')]$. With that, we can say that the mixup label is noisy when the two assignments disagree, i.e. when $\tilde{Y}^* \neq \tilde{Y}$.
  Finally, Theorem 5.1 in Liu et al. [A] gives us $P(\tilde{Y}^* \neq \tilde{Y} | \tilde{X}) \geq TV(P(\tilde{Y} | \tilde{X}), P(Y | X))$, where $TV(\cdot, \cdot)$ is the Total Variation. We can then derive from Lemma D.1 in Liu et al. [A]:
  $TV(P(\tilde{Y} | \tilde{X}), P(Y | X)) = \frac{1}{2} \sum_{m=1}^{M} | P(Y = m | X) - P(\tilde{Y} = m | \tilde{X}) | = \frac{1}{2} \sum_{m=1}^{M} | f_m(X) - ((1-\lambda) f_m(X) + \lambda f_m(X')) | = \lambda TV(P(Y | X), P(Y' | X'))$
  And symetrically:
  $TV(P(\tilde{Y} | \tilde{X}), P(Y' | X')) = \frac{1}{2} \sum_{m=1}^{M} | P(Y' = m | X') - P(\tilde{Y} = m | \tilde{X}) | = \frac{1}{2} \sum_{m=1}^{M} | f_m(X') - ((1-\lambda) f_m(X) + \lambda f_m(X')) | = (1-\lambda) TV(P(Y | X), P(Y' | X'))$
  Now, we come back to our classification problem with $M$ classes, where samples $(x, y), (x', y') \in (\mathbb{X} \times \{1, \dots, M\})^2$ are drawn from probability measures $\nu_{y}$ respectively supported on the class manifolds $\mathcal{M}\_{y}$. Then, $TV(P(Y | X=x), P(Y' | X'=x'))= TV(\mathcal{M}\_{y}, \mathcal{M}\_{y'})$.
  We can thus formally state that: $P(\tilde{Y}^* \neq \tilde{Y} | \tilde{X}) \geq \min (\lambda, (1-\lambda)) TV(\mathcal{M}\_{y}, \mathcal{M}\_{y'})$.
  From all of that we can derive the following insights:

  • i) The probability of assigning a noisy label is lower bounded by the distance (total variation) between the original manifolds. This gives a theoretical justification for the experiments in Figure 1, since the distance between two points roughly informs about the distance between the two manifolds.
  • (ii) To avoid noisy labels, we want to reduce the probability that $\lambda = 0.5$ when the original manifolds are far from each other, i.e., when the distance between the two points is high, which is exactly what our SK Mixup is doing.

• W2 + W3: We thank the reviewer for the references. We will add the discussions in the related work section.

  • AugMix [1] improves robustness by combining multiple image augmentations, which makes the approach limited to image data. Furthermore, RegMixup [B] outperforms AugMix, and we show that we can combine our approach with RegMixup to achieve even better robustness results.
  • NFM and NoisyMix [2,3] both improve robustness by introducing noise before the mixing operations. We think we could combine our method with these approaches, similarly to RegMixup, since the difference seems to not be conflicting. We will investigate that for future work.

  We compared all the methods on ImageNet, ImageNet-R and ImageNet-A in Table 4. We can include more variations of Imagenet for OOD evaluation if the reviewer thinks it is necessary.

• Q1: We presented Eq. 3 that way to clearly show that the Gaussian kernel is centered. When the distance between the two points is equal to the average distance, then the interpolation coefficient is sampled from a $`Beta`(\tau_{max}, \tau_{max})$ distribution.

• Q2: In a high level, the distance between two points from two different manifolds roughly informs about the average pairwise distance between the two manifolds. If the noise applied is the same for all samples (like a gaussian noise with fixed mean and variance), and not too high to avoid manifolds intersecting each other, then we expect the method to keep the same benefits.

[A] Liu et al. "Over-training with mixup may hurt generalization." ICLR, 2023

[B] Pinto et al. "Regmixup: Mixup as a regularizer can surprisingly improve accuracy and out distibution robustness." NeurIPS 2022.

We thank the reviewer for the review and the detailed comments. We answer point by point below.

• W1: We took into account all remarks and improved the presentation of the paper.

• W2: We clarify the connections, from the discussions and results in the paper:

  • We define manifold mismatch as events where mixed samples are outside their original class manifold. Being outside the class manifold can lead to label noise or manifold intrusion. Thus, events of "label noise for a mixed sample" are included in events of "manifold mismatch for a mixed sample".
  • Assignment to a wrong label with mixup is linked to distance between points (Figure 1a in the revision), the higher the distance between points, the higher the chance of the mixed sample being associated to a wrong label. These results are also supported by theoretical grounding, that we introduce as a corollary of Theorem 5.1 in Liu et al. [A], which proves that the probability of assigning a noisy label with mixup is lower bounded by the distance between the two manifold. We refer the reviewer to our answer to reviewer XhfP for the full derivation and discussion (Theorem 3.2 in the revision).
  • Mixing points with lower distance is linked to better calibration (figure 1b), we explain this as mixing only points that are similar reduces occurrence of assigning noisy labels to mixed samples. This improves the quality of labels seen by the model, and its confidence.
  • We show that SK Mixup reduces likelihood of association to a wrong manifold even for points far away (Figure 2.b), compared to original Mixup.
  • We show throughout the paper with many experiments on multiple datasets that SK Mixup improves calibration.

  We hope this summary clarifies the connections between manifold mismatch, association to wrong label, distance between points and calibration.

• W3: We can directly see in Figure 1a that, empirically, the distance between the two points will impact negatively the likelihood of assigning the correct synthetic label to a mixed sample. We also introduce an additional theoretical result supporting this (Theorem 3.2 in the revision).
  While the k-mixup method does take into account the distance between two batch of data, it is to select the points to mix with, which is not the same as "forcing more mixtures between closer points". This method would be more related to C-Mixup. As we discuss in the paper (related work section and in the end of Section 3.1), restricting pair to mix by selecting data based on distance can degrade predictive performance (accuracy, MSE, etc ...) by reducing the diversity of the possible mixing directions and thus the diversity of synthetic samples. We will include this discussion of this reference in the related work section.

• W4: We discuss relation with this work in the related work section (section 2.1). We will reiterate and deepen the discussion. The goal of AdaMixup is to learn the distribution from which mixing coefficient are sampled, and the method is doing so with two external models that predicts the mixing policy to apply depending on each pair of samples. However, as shown in the original paper, the mixing policy is very stable and narrow around 0.5, which means that the predicted mixing coefficients are always very close to 0.5, and that restricts the diversity of synthetic samples. The method is thus fundamentally different from ours, since we explicitly define the mixing distribution as a direct function of the distance.

• Q1-Q4 + Q6: We usually referred to "performance" as a general term for "predictive performance" like accuracy in classification, RMSE or MAPE in regression. We will carefully check the paper to make that clearer.

• Q5: Label noise can be briefly describes as occurring for a mixed sample when its ground-truth assignment and its mixed label does not match, i.e. when the mixed sample should be associated to a different class than the original ones. A more formal definition of label noise can be found in Liu et al. [A].

• Q7: We did not find any reference combining calibration in regression tasks and Mixup.

• Q8: Thank you, we fixed that.

• Q9 + Q10: We wanted to be concise since these are typical notations and settings used in the literature. We explicitly say that M is the number of classes in line 192, but we will clarify the definition of the notations with the two following small modifications:

  • for classifications tasks, $\mathbb{Y} = \{ 1,\dots, M \}$, where $M$ is the number of classes,
  • specify $h_{\varphi}: \mathbb{X} \rightarrow \mathbb{R}^{d_2}$ and $\mathbf{w} \in \mathbb{R}^{d_2 \times M}$, with $d_2$ the embedding dimension, such that $\hat{y} := \arg \max \text{softmax}(f_\theta(x))$, and $f_\theta(x) = \mathbf{w}^T h_{\varphi}(x) \in \{1,\dots, M\}$.

  These do not change the remaining notations in the paper.

• Q11: Yes.

We thank the reviewer for the quick reply. We are glad to know that the reviewer appreciates our changes on the presentation and that our idea is meaningful. However, it is not clear to us what more justifications the reviewer is expecting.

The discussion of the connection between the distances of the example pairs and the occurence of manifold mismatch still seems to be mostly based on intuitions.

The discussion is not "mostly based on intuition", we have two theoretical results supporting it. We prove: (i) the existence of manifold mismatch for any pair of manifold (theorem 3.1) and (ii) the connections between wrong label association (which is linked to manifold mismatch) and distance between manifolds (theorem 3.2). We also present a wide range of experiments showing that taking distance into account improves calibration from state of the art.

In my opinion, this claim itself, is intuitively obvious. One of the other reasons that I mentioned k-mixup is that I am trying to say that this claim is not novel, in fact, it's easy to be noticed.

To clarify, the claim in our paper is that the distance between pairs of points impacts the likelihood of assigning a wrong label with mixup and that it impacts calibration. This is supported in our paper with theoretical and empirical evidence, both being novel results with a novel approach. Furthermore, the theoretical derivations we show in the paper are more general and with different goal than the ones of k-mixup. k-mixup specifically focus on the regularization effect of mixing samples matched with optimal transport, while we study the impact of random matching on the label quality. So our results are also novel in that aspect.

Therefore, if one wants to take this claim as one of the main contributions of their paper, I would prefer to see some more strict justifications. For example, one may mathematically quantify the manifold mismatch behavior and show how the distances of the example pairs may impact this quantity.

We show this in theorem 3.2, we "mathematically quantify" that the probability of associating a wrong label with mixup is lower bounded by the distance between original class manifolds. While the result in theorem 3.2 is using the total variation (TV) between the two manifolds, we can further lower bound the TV by the Wasserstein distance between the two manifolds through Theorem 4 in Gibbs and Su (2002) [A], which is more informative of the distance between points within the manifolds.

We also want to clarify that since mixup is applied for multiple iterations (all batch of data seen during training), using random samplings both to determine which data will be mixed and the weights of the interpolations, its effect is averaged over all iterations. That is why we have to study the impact of mixup in terms of probability and distributions.

[A] Gibbs & Su (2002). "On choosing and bounding probability metrics." International statistical review.

First, the toy example with 3 classes still doesn't fit the claim. Suppose the three classes are three separated line segments on the horizontal axis. If we take: (a) the righest point of class 1 and leftest point of class 2, and (b) the leftest point of class 1 and rightest point of class 3. In this context, case (a) still have much higher chance of having mixed point near original manifolds than case (b).

To be sure that we are on the same page, we considered that in the example, class 1 have been associated to the leftest manifold, class 2 to the middle one, and class 3 to the rightest. Then, that is exactly what we state in the paper ("the higher the distance between two points, the less likely their convex combination will fall near the original manifolds", l. 238 in the revision). The distance in case (a) is lower than the distance in case (b), and the likelihood of the mixed combination being closer to the original manifolds is higher in case (a) than case (b). For case (b) there is even the possibility to create manifold intrusion, since the mixed sample can fall in class 2, and there is also a portion of the line segment that is closer to class 2 than class 1 or 3. As we stated before, the distance between two points roughly informs about the distance between manifolds. Here, the distance in case (a) being lower than the distance in case (b) tells us that manifolds of class 1 and class 2 are closer than manifolds of class 1 and class 3.

Second, I personally believe that it's never "limited by-design" to consider some unideal or even worst-case examples of data distribution in algorithm design in general.

There might be a misunderstanding here, we said that the example was "limited by-design" as it had only 2 classes. Conflicts between class manifolds and label associations during mixup, would require either at least 3 class manifolds or convex hulls of manifolds intersecting. We can add $M > 2$ if the reviewer feels that it is necessary.

We thank the reviewer for their review and appreciate the kind feedback. We answer below to your questions and remarks.

• Q1: There are no inherent problems to apply calibration-driven mixup methods to ViTs, the only limitation is the computation time to have a full benchmark. Here is a more complete table of results for the ViT-S/16 architecture, on ImageNet (IN), ImageNet-R (IN-R) and ImageNet-A (IN-A), that we will include in the revision:

  Methods	IN Acc	IN ECE	IN AECE	IN-R Acc	IN-R ECE	IN-R AECE	IN-A Acc	IN-A ECE	IN-A AECE
  ERM	69.34	9.03	9.03	15.46	35.93	35.93	1.83	45.03	45.03
  Mixup	72.0	5.81	5.8	18.21	29.29	29.29	2.47	41.07	41.07
  Manifold Mixup	72.04	6.99	6.95	18.93	30.27	30.27	2.15	43.47	43.47
  RegMixup	74.44	7.36	7.34	21.01	32.64	32.64	3.8	44.4	44.4
  RankMixup	69.99	9.3	9.3	15.77	37.37	37.37	1.77	46.82	46.82
  MIT-A	72.81	5.69	5.69	17.6	31.07	31.07	2.81	41.8	41.8
  SK-Mixup (Ours)	71.83	3.89	3.9	18.37	28.56	28.56	2.28	39.33	39.33
  SK RegMixup (Ours)	75.11	2.0	1.98	22.06	26.23	26.23	4.44	38.25	38.25

• Q2: This is partially correct. We wanted to separate the definitions of $\lambda_1$ and $\lambda_2$ for clarity, but we overlooked that they need to be defined jointly. We will change this part in the proof by defining both jointly: "On the line segment $[x_k, x_l]$, since $z \in \mathcal{M}_i$ and manifolds are disjoint , $\exists (\lambda_1, \lambda_2) \in [0,1]^2, \lambda_1 > \lambda_2$ such that $\forall \lambda \in ]\lambda_2, \lambda_1[, \tilde{x}(\lambda) = \lambda x_k + (1-\lambda) x_l \notin \mathcal{M}_j$. Then, $\lambda_0 \in ]\lambda_2,\lambda_1[$."

• In Table 4, is there any reason why ECE and AECE are exactly same in OOD settings?

  The only difference between the computations of ECE and AECE is the separation into the bins. AECE dynamically defines the range of the bins to have evenly distributed number of samples within each bin, while ECE fixes the range of each bin. Thus, if the bins are already close to evenly distributed, ECE and AECE will give similar results. We think that it might be more likely to happen in OOD settings. We refer the reviewer to Table 10 in the appendix, reporting AECE in ID settings, in which the values are different than ECE.

• Table 6 does not provide the efficiency of the SK RegMixup in terms of computational cost.

  Here is a more complete comparison of computational costs for baselines considered for Image Classification. We will include these in the table in the revision of the paper.