Dist Loss: Enhancing Regression in Few-Shot Region through Distribution Distance Constraint

We sincerely appreciate your thoughtful feedback, which has allowed us to clarify important aspects of our study. Below, we address the questions in detail:

The form of the loss function $L(.)$

The form of $L(.)$ can be any loss function designed to measure the difference between two sequences based on the application requirements. In this study, we adopt an inverse probability-weighted (INV) L2 loss function as $L(.)$. Assuming the occurrence probability of a sample is $P_i$, the loss weight for that sample is $\frac{1}{P_i}$. The formula for the inverse probability-weighted L2 loss is as follows:

$$\text{INV-L2 Loss} = \frac{1}{N} \sum_{i=1}^{N} \frac{(y_i - \hat{y}_i)^2}{P_i}.$$

It is important to emphasize that previous studies, such as [a] and [b], have shown that solely relying on inverse probability-weighted loss functions is insufficient to address imbalanced regression problems. This limitation is why we did not include this approach as a baseline in our comparison (as the methods compared in this paper show better results) and underscores the vital role of distributional information in Dist Loss. By incorporating this information, Dist Loss more effectively captures and mitigates the discrepancies between the prediction and label distributions, thereby discouraging the model from over-predicting few-shot samples into the many-shot region. To illustrate the computation process of the loss between pseudo-labels and pseudo-predictions, consider the following example: Given pseudo-labels [1, 3, 6] and pseudo-predictions [1, 2, 7], the loss is computed in a standard manner: for each pair of elements (e.g., (1, 1), (3, 2), (6, 7)) at corresponding indices, the loss is calculated, and then the average of these values is taken.

Discretization of the label space and determination of the number of bins

The discretization of the label space in this paper serves to facilitate the use of label distribution information without altering the essence of the regression problem (e.g., transforming it into a classification problem). In practical computation, continuous probability density curves are challenging to process directly; therefore, they are discretized into bins to estimate and utilize the probability distribution.

The number of bins depends on the resolution of the labels relevant to the specific problem. For example, the resolution for age could be 1 year, for blood potassium concentration it might be 0.1 mmol/L, and for temperature, it could be 0.1°C. Once the resolution is determined, the number of bins can be calculated accordingly. For instance, with a bin width of 1 for age, the smallest possible value is 0 and the largest is 150, resulting in 151 bins. In practice, some label values may not occur (e.g., 150 years of age), but this does not impact the task, as we are merely defining a sufficiently large range to accommodate all possible labels.

We hope this explanation clarifies your concerns and thank you again for providing the opportunity to elaborate on these points. Your feedback is invaluable for refining our work.

References

[a] Yang, Yuzhe, et al. "Delving into deep imbalanced regression." International conference on machine learning. PMLR, 2021.

[b] Ren, Jiawei, et al. "Balanced mse for imbalanced visual regression." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

We sincerely appreciate your thoughtful review and insightful suggestions! Below, we have provided a comprehensive response to your questions and comments. If any of our responses fail to sufficiently address your concerns, please inform us, and we will promptly follow up.

Diversity of datasets and models

We sincerely appreciate your valuable feedback. The diversity of datasets and models is indeed crucial for validating the effectiveness of our study.

We would like to clarify that the datasets used in this study include both image datasets (IMDB-WIKI-DIR and AgeDB-DIR) and a time-series dataset (ECG-Ka-DIR), rather than relying solely on image data. Regarding the model architecture, we adopted ResNet-50 for the image datasets, following prior related work. For the time-series dataset ECG-Ka-DIR, we employed a ResNet-like architecture due to its widespread use in ECG analysis. It is important to emphasize that Dist Loss is a model-agnostic approach, meaning its performance is unlikely to vary significantly with changes in the model architecture.

To further enhance the diversity of datasets and models, we constructed a tabular dataset, ECG-Ka-Tab-DIR, based on the ECG-Ka-DIR dataset. Each column in this tabular dataset represents handcrafted features extracted from the original ECG signals (e.g., heart rate). We conducted experiments on this dataset using a two-layer multilayer perceptron (MLP) as the model. The results, presented in the table below, demonstrate that Dist Loss achieves superior performance, excelling in both overall metrics and few-shot scenarios. (Unfortunately, due to technical challenges, we were unable to provide results for FDS on this dataset. We acknowledge this limitation and appreciate your understanding.)

Table 1: Performance comparison on the ECG-Ka-Tab-DIR dataset

Performance under different imbalance ratios

We greatly appreciate and thank you for your insightful comments. To further validate the effectiveness of Dist Loss, we evaluated its performance on the ECG-Ka-DIR dataset under varying imbalance ratios. The results, presented in the table below, demonstrate that our method achieved the best performance in six out of eight datasets (denoted as $D_i$, where $i$ represents the dataset index) and ranked second in the remaining two cases. These findings highlight the robustness of Dist Loss across different levels of data imbalance. For additional details, the corresponding data distribution diagrams are included in Figure 4 of the appendix in the updated version of the paper.

Tabel 2: MAE (Few-shot) performance across different datasets $D_1$ to $D_8$ with varying imbalance ratios

Definition of INV- and objectives of the ablation study on loss functions $L(.)$

"Probability-based inversely weighted (INV)" refers to adjusting the contribution of each sample to the loss based on the inverse of its occurrence probability. Specifically, if a sample's occurrence probability is $P_i$, its loss is weighted by $1/P_i$.

For example, consider L1 Loss, which is typically defined as:

$$\text{L1 Loss} = \frac{1}{N} \sum_{i=1}^{N} |y_i - \hat{y}_i|$$

For INV-L1 Loss, the expression is modified as follows:

$$\text{INV-L1 Loss} = \frac{1}{N} \sum_{i=1}^{N} \frac{|y_i - \hat{y}_i|}{P_i}$$

As mentioned in the paper, Dist Loss measures the differences between pseudo-labels and pseudo-predictions, as well as between true labels and model predictions, using a loss function $L(.)$. This loss function $L(.)$ can be flexibly chosen based on the application requirements. In Table 4 (and the corresponding appendix), we conduct ablation studies to illustrate the effects of using different $L(.)$ functions. For example, INV-L2 generally emphasizes accuracy in the few-shot region due to its higher sensitivity to large errors, making it more effective when the focus is on improving performance in under-represented areas. On the other hand, INV-L1 achieves a balance between overall accuracy and few-shot region accuracy (with a greater emphasis on the latter), making it suitable for scenarios where both aspects are crucial. This highlights the flexibility of Dist Loss, allowing practitioners to design $L(.)$ based on specific application scenarios to achieve tailored performance improvements. However, it is important to emphasize that previous studies, such as [a] and [b], have demonstrated that solely relying on inverse probability-weighted loss functions is insufficient to effectively address imbalanced regression problems. This is why we incorporate distributional information in Dist Loss, which helps to more effectively capture and mitigate the discrepancies between the prediction and label distributions.

We sincerely appreciate your insightful feedback and are happy to provide these clarifications. If you have any further questions, we are happy to address them promptly.

References

[a] Yang, Yuzhe, et al. "Delving into deep imbalanced regression." International conference on machine learning. PMLR, 2021.

[b] Ren, Jiawei, et al. "Balanced mse for imbalanced visual regression." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

How KDE is applied

We apologize for any confusion and appreciate your valuable feedback. Below, we provide a detailed explanation of how Kernel Density Estimation (KDE) is applied in our method:

1. Purpose of KDE: KDE is a non-parametric method used to estimate the probability density function of a random variable without assuming a specific distribution form. In our method, we apply KDE to the labels in the entire training dataset to estimate the probability density of the label distributions.

2. Input and Output: The input to KDE is the set of all labels in the training dataset, and the output is a continuous probability density function of the labels. For simplification, we discretize this continuous density function into $B$ bins (intervals), transforming it into a discrete probability distribution. This discretization is employed solely to facilitate the utilization of label distribution information and does not impact the normal training process.

3. Pseudo-label Generation at the Batch Level: For each training batch, we calculate the expected occurrence count $n_i$ of each label $y_i$ based on the label distribution. This is done to align with the frequency of parameter updates in the model, using the formula: $n_i = \text{batch size} \times P(y_i)$, where $P(y_i)$ is the probability of label $y_i$ in the discretized label distribution. Since $n_i$ is usually a decimal, we round it down to the nearest integer. However, this rounding may cause the sum of $n_i$ values to deviate from the batch size. To address this, we adjust $n_i$ values using Equation (1) to ensure the adjusted values $n_i'$ sum up to the batch size. The final set of $n_i'$ values is denoted as $N_L'$.

4. Construction of Pseudo-labels: Using the adjusted $n_i'$ values, we generate pseudo-labels. For example, if the label set is [1, 2, 3] and the adjusted counts are [1, 3, 2], we repeat label $1$ once, label $2$ three times, and label $3$ twice, resulting in the pseudo-label set [1, 2, 2, 2, 3, 3]. This process is formally described in Equation (2), which defines the mapping between pseudo-label indices and label values.

5. How Pseudo-labels Capture Label Distribution Information: The pseudo-labels generated in this way approximate the original label distribution. If we plot a histogram of the pseudo-labels, it will closely resemble the discretized label probability distribution. This process effectively captures the label distribution information, which is then used for supervised training of the model.

6. KDE in the Testing Phase: It is important to note that KDE is only used in the training phase for pseudo-label construction. During the testing phase, the model is evaluated directly on the test data predictions, without utilizing KDE.

We hope this explanation clarifies the role and implementation of KDE in our method. If you have further questions or need additional clarifications, we are happy to address them promptly.

We sincerely appreciate your thoughtful review and insightful suggestions! Below, we have provided a comprehensive response to your questions and comments. If any of our responses fail to sufficiently address your concerns, please inform us, and we will promptly follow up.

The core idea of Dist Loss and its utility for imbalanced regression

We sincerely apologize for any confusion and appreciate the opportunity to clarify the core concept behind Dist Loss. As illustrated in Figure 1, a primary challenge in imbalanced regression is the mismatch between the predicted value distribution (prediction distribution) and the true label distribution. This discrepancy arises because the model, when minimizing overall error, tends to over-predict samples from the few-shot region into the many-shot region. While this reduces overall error, it severely impacts prediction accuracy in the few-shot region. To address this, we propose Dist Loss, which explicitly minimizes the distance between the prediction distribution and the label distribution during optimization. This approach discourages the model from over-predicting few-shot samples into the many-shot region, thus improving accuracy in the few-shot region. The implementation of Dist Loss introduces an additional loss term that quantifies the distance between the label distribution and the prediction distribution, complementing the standard model optimization. This involves constructing two key components: pseudo-labels and pseudo-predictions, which represent the distributional information of the true labels and predicted values, respectively. The distance between these components is calculated using a loss function $L(\cdot)$, which approximates the difference between the label and prediction distributions. Additionally, the same loss function $L(\cdot)$ is used in the standard optimization process to compute the difference between individual predictions and their corresponding true labels. By combining these terms, Dist Loss aligns the prediction distribution with the label distribution, reducing errors in the few-shot region while maintaining strong overall performance.

We hope this explanation clarifies the motivation and mechanism behind Dist Loss. Please let us know if further details are required.

Comparison with variational imbalanced regression (VIR)

Thank you for your valuable suggestion regarding the comparison with VIR. We added experimental results for VIR on the IMDB-WIKI-DIR and AgeDB-DIR datasets, as used in the original VIR paper, shown in the table below. Our method outperforms VIR in the few-shot region, achieving MAE and GM values of 9.122 and 5.453 on AgeDB-DIR (VIR: 10.427, 7.237), and 22.516 and 13.752 on IMDB-WIKI-DIR (VIR: 23.483, 15.857). Additionally, combining VIR with Dist Loss yields even better results: on AgeDB-DIR, the MAE and GM are 8.159 and 5.145; on IMDB-WIKI-DIR, they are 21.908 and 13.006. This highlights the complementary advantage of Dist Loss in improving existing methods.

Table 2: Comparison with VIR on the IMDB-WIKI-DIR and AgeDB-DIR datasets

Why the overall/many/median/few-shot region setup was not used

1. Detailed Results in the Appendix: We appreciate your insightful concern. As noted in the paper, we have included detailed results for the overall, many-shot, median-shot, and few-shot regions for each dataset in the appendix. These results demonstrate that our method not only maintains overall model performance but also significantly improves performance in the few-shot (and median-shot) regions.

2. Focus on the Few-shot Region: As emphasized throughout the paper, our primary focus is on improving prediction accuracy in the few-shot region, as samples in this region often have more substantial real-world implications. This aligns with the core objectives of imbalanced regression research, as demonstrated in studies [a], [b], and [c]. For instance, in ECG-based potassium concentration prediction, samples corresponding to hyperkalemia and hypokalemia typically fall within the few-shot region (as illustrated in Figure 1 of the paper). Predicting these rare but critical cases accurately is essential in clinical contexts, as abnormal potassium levels can lead to life-threatening conditions such as ventricular fibrillation or cardiac arrest. In contrast, the prediction accuracy of samples within the normal potassium range (many-shot region) is less practically significant, as it does not result in clinically meaningful consequences.

Why the experimental results of ConR differ from the original paper

1. Differences in Experimental Setup: We appreciate your observation regarding the differences in experimental results for ConR. As outlined in the Appendix of our paper, we followed the experimental settings of the original ConR paper and other related works in all respects, with the exception of the number of GPUs used and the number of training epochs. Specifically:

1.1. Number of GPUs: The original ConR paper utilized 4 GPUs in PyTorch's data parallel mode, while we conducted all our experiments using a single GPU.

1.2. Number of Training Epochs: The original ConR paper trained for 120 epochs, whereas other works, such as LDS, FDS, and RankSim, trained for 90 epochs. To maintain consistency across all methods, we standardized the number of training epochs to 90.

2. Consistency with Open-source Code: Furthermore, we utilized the open-source code provided by previous studies (available on GitHub) for our experiments, without any further modifications to the experimental settings. We want to emphasize that no intentional alterations were made to the code that could affect the fairness of our comparisons.

References

[a]. Branco, P., Torgo, L., & Ribeiro, R. P. (2017, October). SMOGN: a pre-processing approach for imbalanced regression. In First international workshop on learning with imbalanced domains: Theory and applications. PMLR.

[b]. Moniz, N., Ribeiro, R., Cerqueira, V., & Chawla, N. (2018, October). Smoteboost for regression: Improving the prediction of extreme values. In 2018 IEEE 5th international conference on data science and advanced analytics (DSAA). IEEE.

[c]. Steininger, M., Kobs, K., Davidson, P., Krause, A., & Hotho, A. (2021). Density-based weighting for imbalanced regression. Machine Learning, 110, 2187-2211.

Thank you for your constructive review and valuable suggestions! Below, we provide a detailed response to your questions and comments. If any of our responses fail to sufficiently address your concerns, please inform us, and we will promptly follow up.

Why methods for imbalanced classification are not well-suited for imbalanced regression

We appreciate your thoughtful comments regarding the challenges of imbalanced regression tasks. While it is possible to transform regression problems into classification tasks by discretizing the regression labels, we argue that this approach introduces several limitations. Below, we outline the key reasons why methods for imbalanced classification are not directly applicable to imbalanced regression:

1. Nature of Regression vs. Classification: Classification tasks have well-defined category boundaries, whereas regression tasks involve continuous and unbounded labels. Discretization disrupts the continuity inherent to regression tasks.

2. Semantic Meaning of Label Distances: The distances between labels in regression tasks carry meaningful information that can facilitate better representation learning. For instance, in age prediction, samples with a label of 31 may be sparse, but the neighboring samples with labels 30 and 32 might be abundant. Consequently, the model's prediction error for 31-year-old individuals can be reduced by leveraging the information from adjacent labels. In contrast, classification tasks treat category values as arbitrary identifiers without intrinsic meaning, preventing the model from utilizing information from neighboring samples, which hinders effective learning.

3. Purpose of Discretization in Our Approach: It is important to emphasize that in our work, discretizing the label space is intended solely to facilitate the utilization of label distribution information, without altering the fundamental nature of the regression problem (e.g., converting it into a classification problem). In practice, continuous probability density curves are challenging to process directly. Thus, dividing the label space into bins offers a practical approach for estimating and utilizing the probability distribution.

Comparative performance analysis with Balanced MSE

1. Overall Performance: We appreciate your insightful comments and the opportunity to discuss the relative performance of our method. While Balanced MSE performs well in the few-shot region, our method outperforms it overall, achieving better results in 5 out of 6 evaluation metrics across 3 datasets. For example, in terms of MAE, we observe relative improvements of 4.42%, 5.11%, and 6.21% on the IMDB-WIKI-DIR, AgeDB-DIR, and ECG-Ka-DIR datasets, respectively. Additionally, a Wilcoxon signed-rank test confirms that our method is statistically superior, with p-values of 0.01, 0.02, and 6.95e-44 for the three datasets.

2. On GM Metric for IMDB-WIKI-DIR Dataset: We observe that Dist Loss achieves a higher GM metric than Balanced MSE on the IMDB-WIKI-DIR dataset. To explore this, we plotted the sorted error distribution curves for both methods in the few-shot region (Figure 5 of the updated paper). While Dist Loss generally outperforms Balanced MSE, small discrepancies around error ranks 5 and 30 may contribute to its slightly inferior GM performance. This is because the GM metric amplifies the effect of small frequent errors, unlike MAE, which averages them. For instance, the GM of (40, 10.1, 10.1, 10.1, 10.1, 10.1) exceeds that of (42, 10, 10, 10, 10, 10), despite the former having a lower MAE. This highlights how frequent small errors can disproportionately impact the GM metric.

3. Additional Metrics: We expanded our analysis by incorporating additional evaluation metrics for the IMDB-WIKI-DIR dataset. Specifically, we transformed the regression problem into a three-class classification task to verify whether the model's predicted values fall within the same region (many-shot, median-shot, or few-shot) as the true labels. We categorized the predictions into these regions and evaluated performance using accuracy and F1-score for each region. Furthermore, we computed the Pearson correlation coefficient to assess the alignment of predicted values with the true labels. The results from these additional metrics indicate that our method consistently outperforms Balanced MSE, demonstrating that our predictions are more accurately aligned with the true label regions across all evaluation criteria.

Table 1: Performance comparison for IMDB-WIKI-DIR dataset (Few-shot region)