Do we need rebalancing strategies? A theoretical and empirical study around SMOTE and its variants

We thank you for your positive feedback, both regarding the theoretical analysis and the experimental section.

You suggest applying our analysis to further data sets with different characteristics. Since we consider only tabular data sets in our study, it is difficult to consider input variable of different types (such as pixels or word embedding). We only consider continuous features as SMOTE was originally designed to handle such features. Extending our proposed method to categorical features is not straightforward, as the dependence between continuous and categorical features must be taken into account in the generative process. Thus, we leave this avenue for future works. Besides, all the data sets already included in the paper treats different machine learning problem. Most of them are used by several seminal works on imbalanced data (see [1,2 and 3]) or included in the open source package imbalanced learn (see [4]). Furthermore, in our study, we already analyze 13 data sets and undersample them some of them times in order to lower the imbalance ratio. If you have any suggestions on how to improve our empirical evaluation, we would be happy to incorporate it in our paper.

You suggest that we should include more recent references and baselines in our work. We have implemented and compared the most common rebalancing strategies, contained in the open-source package imbalanced learn [2], that is class-weight, RUS, ROS, NearMiss1, Borderline SMOTE 1, Borderline SMOTE 2, SMOTE. Note that we implemented the Adasyn strategy but did not display the results as they were very similar to that of SMOTE. We have also used logistic regression, random forest and tree boosting as classifiers. Even if it does not fall into the scope of our study, we also implemented long-tail learning strategies (usually used for images and deep learning algorithms, see Table 17). We are open to suggestions if you have some other specific algorithms in mind.

Finally, you ask if we can provide guidelines to choose the grid of CV-SMOTE. We originally choose the grid to test theoretical values, that is fraction of $n$ and $\sqrt{n}$. We also added some very small values for $K$, as surprisingly, those values can perform well in some data sets. However, we did not obtain a clear picture on what should be the preferred values for $K$. As this question is interesting, we will compute the best values for $K$ (the one output by the cross-validation procedure) and add them to the paper. Thank you for this suggestion!

[1] Chawla, N.V., Bowyer, K.W., Hall, L.O. and Kegelmeyer, W.P., 2002. SMOTE: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16, pp.321-357.

[2] Han, H., Wang, W.Y. and Mao, B.H., 2005, August. Borderline-SMOTE: a new over-sampling method in imbalanced data sets learning. In International conference on intelligent computing (pp. 878-887). Berlin, Heidelberg: Springer Berlin Heidelberg.

[3]He, H., Bai, Y., Garcia, E.A. and Li, S., 2008, June. ADASYN: Adaptive synthetic sampling approach for imbalanced learning. In 2008 IEEE international joint conference on neural networks (IEEE world congress on computational intelligence) (pp. 1322-1328). Ieee.

[4]Lemaitre, G., Nogueira, F. and Aridas, C.K., 2017. Imbalanced-learn: A python toolbox to tackle the curse of imbalanced datasets in machine learning. Journal of machine learning research, 18(17), pp.1-5.

We thank you very much for your careful reading and your numerous comments. We acknowledge that our phrasing is somehow imprecise when discussing the impact of our theoretical or empirical results. This was meant to ease the interpretation but, based on your feedback, we note that this can be misleading. We will correct the paper accordingly to avoid confusions.

• Weakness 1 and 6 : The quantity $\bar{C}(Z, X)$ measures how far SMOTE observations $Z_i$ are from the original observations $X_i$. The quantity $\bar{C}(\tilde{X}, X)$ measures how far data $\tilde{X}_i$, distributed as the original observations $X_i$, are from these observations ($X_i$). The metric $\bar{C}(Z, X)/\bar{C}(\tilde{X}, X)$ quantify the ability of the synthetic samples to fill the space in the same way as samples generated with the true distribution.
  • When $\bar{C}(Z, X)/\bar{C}(\tilde{X}, X)$ is close to one, SMOTE observations are comparable to observations $\tilde{X}$, in terms of the metric $\bar{C}$. In this case, data generated via SMOTE behave in the same way as the original data, with respect to the quantity $\bar{C}$.
  • Similarly, when the quantity $\bar{C}(Z, X)/\bar{C}(\tilde{X}, X)$ is close to zero, observations generated via SMOTE are much closer to their central point than observations generated with the true distribution. In this case, this highlights that SMOTE has a tendency to generate data close to the original observations.
  • Conversely, when $\bar{C}(Z, X)/\bar{C}(\tilde{X}, X)$ is larger than one, data generated via SMOTE are further away from their central point than observations generated with the true distribution. In this case, this highlight that diversity of newly created samples is too high.

The statement $(i)$ “This highlights a good behavior of the default setting of SMOTE (K = 5).” (line 180/181) refers to the Theorem 3.2. Theorem 3.2 establishes that samples generated by SMOTE converge in distribution to the original random variable $X$, which is the asymptotic behavior that we expect from a synthetic data-level strategy. However, Theorem 3.5 and Corollary 3.6 show that SMOTE samples are close to the original data, which leads us to conclude to a lack of diversity or to the fact that SMOTE has a tendency to asymptotically copy the original samples.
Figure 1 illustrates this tendency for SMOTE with default value $K=5$. Our statement $(ii)$ from lines 289 to 296 (the one you quote) is a comment of Figure 1. Thus, there is no contradiction between statements $(i)$ and $(ii)$. We will clarify this ambiguity in the new version.

• Weakness 2 : Any data point generated via SMOTE $Z_{K,n}$ is shown to converge in distribution to any original data point $X$, when $K/n$ tends to zero as $n$ tends to infinity. Thus, SMOTE allows us to generate new data points, which are asymptotically distributed as the original samples. As many statistical analyses, we assume that the size of the minority class tends to infinity in order to study the behavior of SMOTE. Though unrealistic, this setting tends to mimic a data set composed of a high number of observations. Note that this does not contradict the imbalanced classification setting, as we can consider an overall data set whose size tend to infinity, with a fixed proportion of minority samples, therefore allowing the size of the minority class to tend to infinity. Practically, there exist highly imbalanced data sets with a large value of $n$. For example, the CreditCard data set, included in our study, has an imbalance ratio of 0.2% and $n=492$. Indeed, in the banking sector, it is common to have several millions of clients and among them only a few thousands make fraudulent transactions, leading to a high value of $n$ and highly imbalanced data.

• Weakness 3 and Weakness 4: It is difficult to quantify what is the right amount of diversity (for SMOTE samples) that help to improve the classifier performances. In fact, it strongly depends on the distribution of the minority class and of the chosen classifier (logistic regression, random forests...). For example, it is not likely that using SMOTE on linearly separable data improves the performance of a logistic regression, as the original logistic regression on the original data separates properly the original samples. If an oracle tells us that generated data should be distant from less than $\alpha$ from original points with probability $0.95$, our result (equation 109 in the proof) would lead us to choose

$

    K = \left( \frac{0.95 \alpha^2 }{36 R^2 2^{1 + 2/d}} \right)^{d/2} n.

$

Studying theoretically the impact of different level of diversity on different classifier is definitely a very interesting research topic.

Furthermore, through our numerical experiments, we show that applying no rebalancing strategy is competitive, even for highly imbalanced data set such as Yeast which have an imbalance ratio of $11\%$ (see Table 5). Even when we lower the imbalance ratio, the None data set remains competitive for the Breast Cancer, Vehicle, Ionosphere and House_16H data set for example. We recall that our data sets are widely used in the tabular imbalanced data community. For example, Vehicle and Ionosphere data sets are used in [3,4]. Thus, we believe that the empirical part of our study highlights an important practical fact: None is competitive, even for several low-imbalanced data sets.

Finally, to answer your question, it is possible to extend our study to the multiclass framework. We provide here simulations on three data sets with multiclass target (some were already included in our study but binarized). We are willing to add more data sets to the final paper if needed. Furthermore, we look at the ROC AUCs of one class versus the rest and finally average them. We also look at the PR AUC averaged over all combinations. We remark that our introduced strategies remain competitive in multiclass setting.

LightGBM ROC AUC computed on three data sets with multiple classes (average of one-vs-rest ROC AUC).

LightGBM PR AUC computed on three data sets with multiple classes (average of one-vs-rest PR AUC).

[1] Elreedy, D., Atiya, A.F. and Kamalov, F., 2024. A theoretical distribution analysis of synthetic minority oversampling technique (SMOTE) for imbalanced learning. Machine Learning, 113(7), pp.4903-4923.

[2]Elreedy, D. and Atiya, A.F., 2019. A comprehensive analysis of synthetic minority oversampling technique (SMOTE) for handling class imbalance. Information Sciences, 505, pp.32-64.

[3] Han, H., Wang, W.Y. and Mao, B.H., 2005, August. Borderline-SMOTE: a new over-sampling method in imbalanced data sets learning. In International conference on intelligent computing (pp. 878-887). Berlin, Heidelberg: Springer Berlin Heidelberg.

[4]He, H., Bai, Y., Garcia, E.A. and Li, S., 2008, June. ADASYN: Adaptive synthetic sampling approach for imbalanced learning. In 2008 IEEE international joint conference on neural networks (IEEE world congress on computational intelligence) (pp. 1322-1328). Ieee.

Thank you for your detailed review.

You noticed that our statement on line 400 is in contradiction with our definition of imbalance ratio. We thank you for your remark. We propose to replace the statement on line 400 by "probably highlighting that the imbalance ratio is not low enough". We have also checked the whole paper and change the phrasing whenever it was necessary.

You state that the novelty of our theoretical results compared to prior existing works is not clear enough. To the best of our knowledge, there is only Elreedy and al. [1] and [2] that study theoretically SMOTE. In [2], the authors derive the expectation and the covariance of the samples generated by SMOTE. Elreedy and al. [1] derive the density of samples generated by SMOTE, and confirm this density with numerical experiments on simulated data. Thus, these two papers derive statistical quantities related to SMOTE algorithms, without drawing any conclusions regarding the performance/drawback of the actual SMOTE algorithm. In our paper, we use their derivations (see Lemma 3.3) to improve our knowledge on SMOTE algorithm. We first prove that SMOTE converges in probability to the minority class distribution as $K/n \to 0$ (Theorem 3.2), based on the concentration of the nearest-neighbors. Then, we proved that conditionally to the central point, an observation generated via SMOTE tends to the central point in probability (Theorem 3.5). Then, Corollary 3.6 gives a characteristic distance to this agglutination phenomenon using only inequality on the density of the nearest-neighbors. In particular, this result is valid in finite-sample settings, where $n$ is not assumed to tend to infinity. In a similar finite-sample setting, we establish in Theorem 3.7 an upper bound of SMOTE density near the boundary of the support of the minority class. This result shows that SMOTE density vanishes near the boundary of the support, thus compromising the regeneration of the minority samples near the border. From those theoretical results, we deduce all our intuitions about SMOTE behavior. Except for Lemma 3.3, all our results are new and thus absent from [1,2]. Besides, Elreedy et al. [1,2] do not use their derivations of SMOTE density in order to study SMOTE behavior.

You share some concerns on the novelty for CV-SMOTE and MGS compared to existing methods. Through our theoretical study, we proved that asymptotically, the default value $K=5$ of SMOTE leads to a lack of diversity in the samples generated by SMOTE.
Then we show that for some values of $K$ (such that $K=0.1n$), SMOTE observations are more diverse. We tried different asymptotics for $K$ (like $K = \sqrt{n}$, $K = \alpha n$ for different values of $\alpha$) but we could not find a universal best choice. Thus, we introduce a grid optimization procedure to automatically choose the best value for $K$. We would like to remark that SMOTE is a widely used algorithm with its default hyperparameter value $K=5$ and to the best of our knowledge there is no paper recommending tuning $K$. MGS is introduced in order to have a process that do not generate new samples only in a vector line. This would lead to more diverse synthetic samples again and a better reconstruction of the minority class distribution. It is surprising that such small modifications of existing algorithms lead to two new procedures with improved performances. Our goal is by no mean to say that these two methods improve drastically over existing ones. Indeed, for most data sets, None method is the best one. Thus, one of our major contribution is to establish that None is among the best strategy and that, in the remaining cases, our two proposals are competitive.

You also explain that it is known that applying no rebalancing strategy performs competitively for mildly imbalanced data sets. We believe that this fact is not well-known in the ML community: many research papers as [3,4] aim at designing new rebalancing strategies to improve upon None. Many papers conclude that their strategy is better than None. For example, SMOTE is shown to outperform None in [3] (in terms of F1-score and True Positive rate). Similarly, borderline-SMOTE strategies outperform None (see page 8 of [3]). Besides, in the Adasyn introduction article [4], it is shown that SMOTE and Adasyn offer significant improvement of predictive performances (in terms of F-measure and G-mean) over None strategy (see page 5 of [4]). Therefore, to the best of our knowledge, no article introducing new oversampling strategies explicitly conclude that None is competitive.

You share some concerns on the assumptions of Theorem 3.2 and Theorem 3.5 ($K$ and $n$ which tend to infinity). Any data point generated via SMOTE $Z_{K,n}$ is shown to converge in distribution to any original data point $X$, when $K/n$ tends to zero as $n$ tends to infinity (Theorem 3.2). Thus, SMOTE allows us to generate new data points, which are asymptotically distributed as the original samples. As many statistical analyses, we assume that the size of the minority class tends to infinity in order to study the behavior of SMOTE. Though unrealistic, this setting tends to mimic a data set composed of a high number of observations. Note that this does not contradict the imbalanced classification setting, as we can consider an overall data set whose size tend to infinity, with a fixed proportion of minority samples, therefore allowing the size of the minority class to tend to infinity. Practically, there exist highly imbalanced data sets with a large value of $n$. For example, the CreditCard data set, included in our study, has an imbalance ratio of 0.2% and $n=492$. Indeed, in the banking sector, it is common to have several millions of clients and among them only a few thousands make fraudulent transactions, leading to a high value of $n$ and highly imbalanced data. Besides, you ask if some of our theoretical results holds for a fixed $n$. Note that Corollary 3.6 is valid for any value of $n$. Besides, Theorem 3.5 and Theorem 3.7 are valid in a finite-sample setting, when $n$ is fixed but larger than a specific value (depending on $K$). Thus, our results are not only asymptotic, but leads to an understanding of SMOTE behavior in finite-sample settings.

We agree that we provide an analysis of SMOTE properties, regardless of the majority class. Our analysis focuses on the ability of SMOTE to generate data according to the true distribution of the minority class, which is a first guarantee for using such a strategy in imbalance learning. We would like to emphasize that our work already leads to important conclusions on SMOTE: default value of $k=5$ should not be used by default, and the ability of SMOTE to generate samples with the correct distribution is damaged near the boundary of the support of the minority class. Including the majority sample (and thus the imbalance setting) in our analysis cannot be done without taking into account the predictive algorithm (logistic regression, random forests...). In this case, our conclusion would depend on the predictive algorithm. We leave to future work to try to find theoretical settings in which one can establish properties of rebalancing strategies. Again, note that previous works on SMOTE simply establish the density of newly generated data points, or compute the expectation or the covariance matrix of the corresponding distribution.

Through our theoretical study, we proved that asymptotically, the default value $K=5$ of SMOTE leads to a lack of diversity in the samples generated by SMOTE.
Then we show that for some values of $K$ (such that $K=0.1n$), SMOTE observations are more diverse. We tried different asymptotics for $K$ (like $K = \sqrt{n}$, $K = \alpha n$ for different values of $\alpha$) but we could not find a universal best choice. Thus, we introduce a grid optimization procedure to automatically choose the best value for $K$. We would like to remark that SMOTE is a widely used algorithm with its default hyperparameter value $K=5$ and to the best of our knowledge there is no paper recommending tuning $K$. MGS is introduced in order to have a process that do not generate new samples only in a vector line. This would lead to more diverse synthetic samples again and a better reconstruction of the minority class distribution. It is surprising that such small modifications of existing algorithms lead to two new procedures with improved performances. Our goal is by no mean to say that these two methods improve drastically over existing ones. Indeed, for most data sets, None method is the best one. Thus, one of our major contribution is to establish that None is among the best strategy and that, in the remaining cases, our two proposals are competitive.