Selective Mixup Helps with Distribution Shifts, But Not (Only) because of Mixup

Thanks for your time. The review seems to use a few incorrect assumptions, e.g. the paper makes zero claims about long-tail scenarios (c.f. comparison with MiSLAS). We propose to clarify the paper with the points below.

Evaluation datasets The authors do not provide results for the imbalanced classification on standard datasets such as CIFAR-10LT, CIFAR-100 LT, and imagenet1k-LT.

• The 5 datasets we use come directly from the most directly-relevant prior work (selective mixup for distribution shifts).

• Our main claim is that this prior work missed important ablations, hence our priority is to perform the missing experiments on the same datasets.

• The point of this paper is to improve our scientific understanding of selective mixup with distribution shifts. Experiments on *-LT could be interesting but would add little value to the main claims.

Could you do an analysis where the test distribution is also skewed, independent of the training distribution? (...)

The test distribution is skewed "independent of the training distribution" (?) as our default setting since the whole paper is about distribution shifts. If the question is specifically about shift of the label distribution, this is exactly what is investigated with the yearbook/MIMIC/arxiv datasets (see Fig. 10; the test distribution is not uniform).

I find a lack of novelty in the finding that mixup yields a training distribution with higher entropy

Not sure what "lack of novelty" means (??). Prior work missed this as an explanation for the effects of selective mixup. This is exactly why this paper is important: it highlighting an effect that has been in plain sight but missed in highly-cited prior work.

Could you provide any strong theoretical justification that the actual mixup is not contributing to an improvement

The paper never makes this claim, which is factually incorrect from our experiments. The paper states the exact opposite multiple times:

• yearbook dataset: "(...) confirms the complementarity of the effects of resampling and within-class selective mixup"
• arxiv dataset: "the performance of selective mixup is explained by cumulative effects of vanilla mixup and resampling effects"
• civilComments dataset: "the performance of selective mixup is the result of the independent effects of vanilla mixup and resampling"

Different datasets benefit differently from mixup and/or resampling. This is why there is no single simple story, and why the results are analyzed on each dataset separately.

Thanks for your time. These comments are very helpful for improving the paper. We propose to clarify the points below in the text, and make this general message clearer upfront: different datasets benefit differently from mixup and/or resampling. This is why there is no single simple story and why each dataset is best analyzed individually.

In Figure 2, the authors mention that ''The ranking of various criteria for selective sampling is similar whether with or without Mixup''. However, claiming that the performance between selective sampling is similar to that of selective Mixup seems somewhat strained. For instance, in the case of "Diff. domain+ Same class," selective Mixup demonstrates a higher accuracy than selective sampling. What accounts for the superiority of selective Mixup over selective sampling in this scenario?

There are indeed small differences (<5%), which we propose to mention in the caption. We will clarify that the key observation is correct for the best-performing version (same domain/diff. class, and resampling for uniform combinations), which is >25% above the baseline and the version that most will care about.

Similarly, in Figure 2, (...) does this indicate that vanilla Mixup is harmful in this case?

Yes indeed. One can see it just by observing that vanilla mixup is worse than the baseline on this dataset. Prior work has indeed shown that mixup is not always beneficial.

The authors have not discussed the reasons behind the occasional superiority of selective Mixup

We already acknowledge that some cases benefit from selective mixup in the way proposed by Yao et al. For example see Fig.4 (yearbook): "it indicates a genuine benefit from mixup restricted to pairs of the same class". This is also exactly what the title of the paper says ("not always"). We propose to highlight this takeaway in the caption of Fig.4.

In Figure 6, given the effective performance of vanilla Mixup, it appears that vanilla Mixup is the main driver for the improvement in selective Mixup, even with the optimal criteria. This observation contradicts the third point outlined in the summarized contributions in the Introduction.

The reviewer is absolutely correct. On this dataset, 4/5 of the improvement over the baseline can be attributed to vanilla mixup, and 1/5 to resampling. We propose to tone down the third summary point ("resampling is SOMETIMES the main driver").

Does this address your concerns? Happy to take other suggestions for improvement.

Thanks for your time. These comments are very helpful by showing that the significance of our findings was not sufficiently highlighted. We propose to clarify the paper with the following points.

novelty / doesn’t provide insights different than existing works (Yao et al. 2023)

On the contrary, this work overturns part of Yao et al. We show that the widely-accepted explanation of their method is incomplete or even incorrect in some cases.

This paper is important because it improves the scientific understanding of this highly-cited method. Incorrect explanations will derail future work that builds upon current knowledge. And getting to such deep understanding is the whole point of science.

only considers the mixup of the labels (...) the mixing of data points (x) should also be considered (i.e. covariates)

This is indeed an interesting question that we openly discuss p.4. The proposed formalization with labels does not readily extend (e.g. a uniform distribution over covariate isn't well defined) but the extensive experiments suggest that the same mechanism is at play with labels and covariates. We propose to make it clear that this is a limitation of our formalization.

The authors explain the results of each of the datasets independently (...) it’s hard to obtain final conclusions

There is a simple conclusion: there is no simple conclusion :)

Different datasets benefit differently from mixup and/or resampling. This is why there is no single simple story and why each dataset is best analyzed individually. We propose to make this message clearer upfront in the paper.

Do these clarfifications address your concerns? We are grateful for your contribution to the improvement of this paper!

Thanks to the authors for the rebuttal. I have read other reviews and rebuttal responses. After this, my major concern regarding the claim of "resampling being the main reason for mixup" (i.e., empirical results scattered) being not supported by sufficient experiments still persists. Further, no revision has been submitted by the authors, which improves the clarity of the results. Hence, at this point, I will maintain my rating and leave further discussion to AC.

Thanks for your time. These comments are very useful. We propose to clarfiy the following points in the paper.

Some of the notations are not so clear. Like in Table 1, what's the definition of Resampling (uniform cl.) + concatenated pairs, and why it has different proportion of majority class compared with "Resampling (uniform classes)"?

What makes the difference in the sampling ratio between the selective sampling without mixup and resampling?

This can indeed use a formal explanation:

• Resampling (uniform cl.) + mixup uses paired instances such as: {$\textrm{mix}(x_1, \tilde{x}_1), \textrm{mix}(x_2, \tilde{x}_2), \textrm{mix}(x_3, \tilde{x}_3), ...$} where $x_i$ are sampled with uniform classes (50% proportion) and $\tilde{x}_i$ are sampled indiscriminatively (hence with the original 78% proportion). Overall we get something inbetween (64%).

• Resampling (uniform cl.) + concatenated pairs uses the same instances as above without the mixing: {$x_1, \tilde{x}_1, x_2, \tilde{x}_2, x_3, \tilde{x}_3, ...$} The proportion of labels is thus also ~64%.

When changing the lambda for the beta distribution of mixup, are similar results as Fig 15 hold?

Yes, we performed experiments on datasets with various $\lambda$ and found very little impact.

For this finding, what about using another mixup method like manifold mixup, which may help the old generalization ability by connecting samples from different domains/classes in the representation space?

Indeed, we follow Yao et al. and use manifold mixup (which is the only sensible option for the NLP and MIMIC tasks). We made this clearer in the text.

Does this properly address your questions? We are grateful for your contribution to the improvement of this paper!