Tailoring Mixup to Data using Kernel Warping functions

We thank the reviewer for their constructive feedbacks on our work.

• W1:

  • 1. Comparison with RegMixup: We would like to refer the reviewer to Appendix B in which we show that compared to both RegMixup and MIT, our proposed approach is much more efficient both in terms of computation time and memory usage. Our method relies on a single batch of augmented data, while the augmentations in RegMixup and MIT double the size of the batch, leading to more compute and more required memory, as discussed in section 4.1 in the paper.
  • 2. Manifold Mixup on classification (Table 1): We reproduced results of Manifold Mixup (Verma et al., ICML 2019) on CIFAR 10 and CIFAR100 using a Resnet50. We achieve similar performance improvements, but with better calibration. As for RegMixup, we can consider combining our method with Manifold Mixup in future work.
  • 3. Data augmentation aims to cover uncertain data points (Table 2): The assumption underlying this work is that there is a trade-off in data augmentation procedures between adding diversity and introducing uncertainty. While we agree that it is important to cover more regions of the space, it is important to do it without adding errors and avoid manifold intrusion (Guo et al., 2019; Baena et al., 2022), i.e, conflicts between the synthetic labels of the mixed-up examples and the labels of original training data. Our intuition is that mixing data that are too far introduces uncertainty on the true underlying label of the new point, for which a linear combination of the labels might not be an accurate target. We confirm this hypothesis by the following experiment that we report in Table 2, in which we compare accuracy and calibration metrics when mixing only pairs of points with distances lower or higher than a given quantile of the overall pairwise distances within the batch, using a Resnet34 on CIFAR10 and CIFAR100 datasets. Note that one should compare results of "Lower $q$" with "Higher $1 - q$" to have equivalent numbers of possible element to mix with (diversity). We can see that, in general, mixing pairs with lower distances leads to better calibration than mixing pairs with higher distances (uncertainty). The introduction of uncertainty when mixing pairs with high distance can be explained by manifold intrusion (Guo et al., 2019; Baena et al., 2022). These results confirm that there is a trade-off between adding diversity and uncertainty with data augmentation.

• W2. This is an interesting remark for which several points need to be discussed.

  • 1. Statistics can be computed before training (Table 3): First, if one can use a meaningful distance between data points different from the distance in the embedding space of the model to train, statistics such as mean or median of the pairwise similarity matrix can be computed beforehand to normalize the distance during training. In our experiments on regression tasks, we also compared with normalizing using the mean over the whole training set instead of the batch. We found slightly improved results on Airfoil, for which the batch size is 16, but no meaningful improvements for the two other datasets, for which the batch size is 128. These results are provided in Table 3.
  • 2. Median can be used to be more robust to outliers: Second, our rationale behind the normalization by the mean is mainly to rescale the distances to similar scales between tasks and models, and same for $\tau_{max}$ and $\tau_{std}$. If the batch size is too small, one could use the median instead of the mean to be more robust to outliers.
  • 3. Distance impact distribution of $\lambda$: Third, regarding the remark "the same samples will be mixup with different $\lambda$ whether an outlier appears", we would like to point that since we are sampling different $\lambda$ at each iteration, it is very likely that the same pair of samples will be mixed with different $\lambda$ at each iteration, whatever the mixup approach we are using. In our case, the underlying distribution of the $\lambda$ might vary for the same pair of samples between iterations depending on the distances in each batch, but we do not think that it hinders the robustness of the approach. If the distance between two samples is high, it is more likely that the $\alpha$ parameter will be small and thus that $\lambda$ will be close to 0 and 1, even if the batch size is small.

• W3. See W1.3 We provide in Table 2 an analysis to back up our intuition that mixing data that are too far introduces uncertainty on the true underlying label of the new point, and that a linear combination of the labels might not be an accurate target.

We thank the reviewer for taking time to review our work and for their constructive feedback.

• (W1) May we ask the reviewer for references linked to the claim on the novelty of our work ? As far as we know, at the time of submission, we did not find related work that also proposed to dynamically change the underlying distribution of $\lambda$ during training. Furthermore, we are not controlling the pairing of points, but how strongly the data are mixed by changing the sampling distribution of the $\lambda$, using the distance between the two points to mix, and not distribution similarities unlike k-mixup (Greenewald et al., 2021).
• (W2) Our goal is to study the impact of mixup on calibration in general for both classification and regression tasks. We considered datasets for classification that are widely used in the literature to accurately compare with other baselines, and they are mainly image datasets. However, it is not the case for the regression tasks, where the datasets used are tabular and times series data. Specifically, on Airfoil (tabular data), we can see that both mixup and manifold mixup have negative effects on performance and calibration, but not with neither C-Mixup nor our approach. In this case, we are using L2 distance between labels since it works best in this case as shown in Yao et al. (2022).
• (W3) While we agree that finding the correct values for $\tau_{max}$ and $\tau_{std}$ through cross validation beforehand takes additional time, we would like to point that it is the case for any hyperparameter and that other mixup methods also have additional hyperparameters, such as $\alpha$ in the Beta distributions or the strength of the regularization in RegMixup, or the layers selected to apply mixup in Manifold Mixup. In our case, we do not have to search for a good potential $\alpha$ since it is already reflected in both $\tau_{max}$ and $\tau_{std}$, which makes one less hyperparameter.
• (Q1) When stating that our method "improves both performance and calibration", we are referring to generalization performance in-distribution, i.e. accuracy on the test set, and that we improve calibration at the same time.
• (Q2) The exponential correlation is linked to the warping function that we consider, i.e. in our case, the Beta CDF function. As can be seen in figure 2 of the paper, there is a logarithmic correlation on the shape of the Beta CDF and the parameter $\tau$. Based on this, we defined the similarity kernel with an exponential correlation with the distance. As mentioned in the paper, the correlation with the distance will depend on the warping function considered in the framework.
• (Q3) We present in the paper several possible distance metrics that we tried, L2 distance between inputs, between embeddings of input, between classification weights of the associated true class or between labels. We also tried Cosine distance instead of L2 distance, but did not observe meaningful difference between the two. However, we agree that one could consider optimal transport distances as well if each data can be seen as a distribution (since we are computing similarity for a pair of data), which could be possible for images using distributions of pixel values.
• (Q4) The cross-validation is conducted before training, with a stratified sampling on a 90/10 split of the training set, and the results are averaged across 4 different splits. The hyperparameters are selected to have a good trade-off between calibration and accuracy. As mentioned in Pinto et al. (2022), cross-validating hyperparameters based solely on the ECE can prefer models with lower accuracy but better calibration. However, a method improving calibration should avoid degrading accuracy. We will clarify the process of the cross validation in the paper.
• (Q5) We thank the reviewer for pointing out this typo in the algorithm. We compute $\tau_i^{(i)}$ with $\tau^{(i)}(\mathbf{x}, i, \sigma; \tau^{(i)}\_{max}, \tau^{(i)}\_{std})$, and $\tau_i^{(o)}$ with $\tau^{(o)}(\mathbf{x}, i, \sigma; \tau^{(o)}\_{max}, \tau^{(o)}\_{std})$. If one wants to use different $\tau^{(i)}$ and $\tau^{(o)}$, one has to use different pairs of $(\tau^{(i)}\_{max}, \tau^{(i)}\_{std})$ and $(\tau^{(o)}\_{max}, \tau^{(o)}\_{std})$.
• (Q6) While we agree that we could have considered input or embedding distance for regression tasks, it was shown in Yao et al. (2022), that label distance works best in this case. We thank the reviewer for this remark and will make it clear in the revised version.

References

• Pinto et al., RegMixup: Mixup as a Regularizer Can Surprisingly Improve Accuracy and Out Distribution Robustness. NeurIPS, 2022
• Yao et al., C-Mixup: Improving generalization in regression. NeurIPS, 2022.
• Greenewald et al., k-Mixup Regularization for Deep Learning via Optimal Transport. arXiv:2106.02933, 2021

We thank the reviewer for their kind comments on our work.

• (W1) The main goal of the paper is to study and improve the impact of mixup on calibration in general. We present experiments for both classification and regression tasks, since it is two very different settings, and not that well studied in regression. Nevertheless, we show the flexibility of our framework, which can improve calibration in both tasks. However, as mentioned in the conclusion of our paper, we are interested to apply our method on more difficult regression tasks such as Monocular Depth Estimation, but leave that to future work.
• (Q1) As presented in Section 3.3, we use an L2 distance, on either input data, embedding of inputs, classification weights or labels depending on the task.

We thank the reviewer for their review and extensive feedback on our work.

• (W1) The assumption underlying this work is that there is a trade-off in data augmentation procedures between adding diversity and introducing uncertainty through manifold intrusion (Guo et al., 2019; Baena et al., 2022), i.e, conflicts between the synthetic labels of the mixed-up examples and the labels of original training data. To show that, we conduct in Table 1 an empirical analysis that compares accuracy and calibration metrics when mixing only pairs of points with distances lower or higher than a given quantile of the overall pairwise distances within the batch, using a Resnet34 on CIFAR10 and CIFAR100 datasets. Note that one should compare results of "Lower $q$" with "Higher $1 - q$" to have equivalent numbers of possible element to mix with (diversity). We can see that, in general, mixing pairs with lower distances leads to better calibration than mixing pairs with higher distances (uncertainty). The introduction of uncertainty when mixing pairs with high distance can be explained by manifold intrusion (Guo et al., 2019; Baena et al., 2022). These results confirm that there is a trade-off between adding diversity and uncertainty with data augmentation.
• (W2) Along with the empirical analysis detailed above, we also provide experiments on the Tiny-Imagenet benchmark in Table 2. We can see that our approach improves both performance and calibration over Mixup baselines on this dataset as well, and reach similar calibration than RegMixup. We are working on adding the other baselines for this dataset as well. As mentioned in the conclusion of the paper, we are working on combining our approach with other mixup methods. Regarding the references mentioned by the reviewer:
  • The OpenMixup paper [1] and library mentioned is still under construction, but we plan to contribute to this library once our paper is accepted.
  • We are already comparing against C-Mixup [2] on the same datasets in the experiments on regression tasks.
  • CutMix [3,5], PuzzleMix [4] and TransMix [6] are methods focused on image data. As our main goal is to study and improve the impact of mixup on calibration in general, we do not want to restrict our evaluation to image datasets and image methods. We consider regression tasks on tabular and time series data, to evaluate calibration of Mixup methods in this setting, which have rarely been investigated in the literature, but all of the above methods are not adapted to these types of data.
  • As both MixupE [8] and Decouple Mixup [7] have been published very recently, implementing these methods will require further work.
• (W3) As mentioned in Section 4.1, we search for the best hyperparameters through cross-validation. The cross-validation is conducted before training, with a stratified sampling on a 90/10 split of the training set, and the results are averaged across 4 different splits. The optimal values will vary between datasets, since they depend on the statistics of the pairwise distances. We assume that the behavior of these parameters is also impacted by the number of class and their separability in each dataset. We provide the resulting heatmaps of the cross validation in Figure 5 and Figure 6 for a better visualization, but we will also include the exact values obtained for all the cross validations in the Appendix.
• (W4) We will make sure to add the latest relevant papers that we might have missed since the moment of submission. Regarding the two subsections mentioned by the reviewer, we will try to combine them, but it might be difficult since they cover different topics. While Sec. 2.2 presents related work on calibration, Sec. 3.1 introduces notations used throughout the paper.

Thank you for taking the time to reply.

(W1)

1. Empirical evidence of the impact of distance on calibration: We have conducted the empirical studies mentioned by the reviewer, in Table 1 of our first answer, to precisely study the impact on calibration of the distance between the points to mix. These experiments bring empirical evidence that, indeed, mixing pairs with lower distances leads to better calibration than mixing pairs with higher distances.
2. On the importance of the contribution: The problem of the trade-off between performance and calibration with Mixup have been extensively studied in previous work (Thulasidasan et al. 2019; Zhang et al., 2022; Wang et al., 2023), as mentioned in Section 2.2. The motivation of our paper is to propose a more efficient approach to achieve a better trade-off in terms of performance and calibration by controlling the impact of distance on mixing.

(W2) We would like to point that what the reviewer calls "small-scale image data, time series data and tabular data", encompasses a lot of possible applications and represents a wider scope than previous Mixup papers limited to image data. During the rebuttal, we have also included experiments on Tiny-ImageNet, which is on a bigger scale than CIFAR. Nothing prevents applying our approach to larger scale, this is something that we are working on, but needs more time.

The paper is now updated with the changes.

References:

• Thulasidasan et al., On mixup training: Improved calibration and predictive uncertainty for deep neural networks. NeurIPS 2019
• Zhang et al., When and how mixup improves calibration. ICML 2022
• Wang et al., On the pitfall of mixup for uncertainty calibration. CVPR 2023

Thanks for the detailed response to my concerns. Sorry for the late reply at the end of the rebuttal period! My concerns about W3 and W4 have been addressed (it is better for the authors to update them in the revision). However, I am not convinced by the replies to the first two issues, and I decided to maintain my rating at the current stage.

• (W1) Despite the authors explaining the motivation and the main problem in this paper, I think some empirical studies are required to demonstrate the studied problem really existed in previous works. This is directly related to the effectiveness of the proposed method. Since the performance gains of the proposed method are not significant or even worse than existing methods, I really need the relevant evidence to support the problems and statements in the authors' response.

• (W2) I have acknowledged the purpose of the proposed methods and the reasons for not conducting the comparison experiments I recommended. This leaves me wondering what is the practical value of the proposed approach if it mainly addresses small-scale image data, time series data, and tabular data. From my perspective, it makes this article's contribution seem inadequate to be accepted by the ICLR community.

Except for the unsolved issues, I appreciate the improved perspective of this article, the comprehensive evaluation metrics, and the overall presentations.

Table 1: Empirical study of the effect of distance when mixing using Resnet34 on C10 and C100