Entropy-Calibrated Label Distribution Learning

Dear Reviewer Z4dW,

Thank you for your positive assessment on our work. We sincerely appreciate the time and effort that you have dedicated to evaluating our work. We have carefully considered all the comments and have revised the paper accordingly. Below, we provide a point-by-point response to your suggestions.

Weakness 1: The experimental configurations of this paper are not adequately shown, such as the specific configuration of the optimizer used and the iteration stopping conditions.

We have added more necessary information about the experimental configurations. The optimizer used in our paper is the L-BFGS algorithm implemented by pytorch. The code is shown as follows:

optimizer = torch.optim.LBFGS(..., lr=1e-3, max_iter=1000, tolerance_grad=1e-5, tolerance_change=1.4901161193847656e-8, history_size=5, line_search_fn='strong_wolfe', max_eval=None)

The above code shows that the optimization process will be stopped when the change of loss function is smaller than 1.4901161193847656e-8 or the gradient is smaller than 1e-5.

Weakness 2: The writing of the paper could be further improved.

We have improved the writing of our paper. For example, we have added a left parentheses before the "Art Painting"; we have added a period at the end of the sentence in line 197; we have added the entropy variance in the caption of Fig. 5(a); we have calibrated the inconsistency of period of the paragraph subheadings.

Question 1: In order to make the anchors as dispersed as possible, why not just minimize the inner product of the anchors instead of minimizing the cosine similarity of the two anchors?

The inner product of anchor vectors is also determined by their norm, which undermines its reliability in preventing over-uniformity. Mathematically, the inner product between anchor vectors equals to their cosine similarity multiplied by their $L_2$ norms. Consequently, the minimization of the inner product could result merely from reduced vector norms rather than improved angular dispersion (i.e., decreased cohesion among anchor vectors).

Question 2: Are there some empirical suggestions for hyperparameter selection?

As analyzed in lines 249-254 of our paper, we recommend $\alpha=40$ as a default setting for most datasets, as it approaches optimal performance across diverse scenarios. For further performance optimization, we suggest the following empirical guidelines:

1. When the training set contains predominantly high-entropy label distributions, consider using $\alpha <40$.
2. For datasets with predominantly low-entropy label distributions, larger values are preferable.

Dear Reviewer zUMv,

Thank you for your positive assessment on our work. We sincerely appreciate the time and effort that you have dedicated to evaluating our work. We have carefully considered all the comments and have revised the paper accordingly. Below, we provide a point-by-point response to your suggestions.

Weakness 1: In line 42, "the samples with high-entropy" should be "the samples with low-entropy".

As suggested, we have corrected this typo error in the revised version.

Weakness 2: For better clarity, it would be beneficial to mention the term "ECA" in Section 4.

As suggested, we have added an introductory paragraph about ECA at the beginning of Section 4 to elucidate its objectives and main concepts. The details are as follows. In this section, we illustrate the proposed ECA (Entropy-Calibrated Aggregation) strategy to address the entropy bias in conventional model performance evaluation methods. Following the divide-and-conquer principle, the main idea underlying ECA is to evaluate the model performance separately on the low-entropy and high-entropy subsets of test set.

Weakness 3: In the main text, it would be beneficial to reference the Appendices explicitly. For example, after presenting Theorem 3.1, a note should be added indicating that its proof is provided in Appendix A.

As suggested, we have added the necessary references of Appendices in the main text. For example, the proof of Theorem 3.1 is provided in Appendix A; the introduction of datasets is provided in Appendix B.1; more experimental analysis are provided in Appendix B.2.

Question 1: What is the form of the function $f (\\cdot)$? Does it use only anchor vectors as classifier weights?

The function $f (\\cdot)$ in the paper is a multivariate function that maps the feature vector $\\boldsymbol{x}$ to the label distribution. Specifically, it uses a softmax normalization over the dot products between the feature vector $\\boldsymbol{x}$ and the anchor vectors $\\{\\boldsymbol \\omega _ m\\} _ \{m=1\}^M$. The form of $f(\\cdot)$ is $softmax([\\langle \\boldsymbol\\omega_m, \\boldsymbol x\\rangle ] _ \{m=1\}^M)$ where $\\langle \\boldsymbol \\omega _ m, \\boldsymbol x \\rangle$ is the dot product between the anchor vector $\\boldsymbol \\omega _ m$ and the feature vector $\\boldsymbol x$, and $softmax(\\cdot)$ ensures the output is a valid probability distribution.

The function $f(\\cdot)$ does not use only anchor vectors as classifier weights. The function $f(\\cdot)$ is theoretically compatible with various representation learning techniques, serving as a classifier when combined with architectures like convolutional neural networks for image classification or graph convolutional networks for node classification. Specifically, in typical label distribution learning (LDL) models, the output can be formulated as $softmax([\\langle \\boldsymbol\\omega _ m, \\boldsymbol v \\rangle ] _ \{m=1\}^M)$, where $\\boldsymbol v$ represents the feature vector of a sample. In deep learning scenarios, $\\boldsymbol v$ is usually derived by passing the raw feature vector $\\boldsymbol x$ through a feature extraction network, whereas in non-deep learning settings, $\\boldsymbol v$ is often directly set to $\\boldsymbol x$. However, in our experiments, we deliberately abstain from incorporating representation learning techniques and instead employ anchor vectors exclusively as classifier weights. This choice is motivated by two key considerations: (1) Most available label distribution datasets are tabular in nature, rendering complex neural network architectures unnecessary. (2) To ensure a fair comparison with state-of-the-art baseline methods, which predominantly do not adopt representation learning techniques, we also avoid a representation learning technique. Thus, our practical implementation of $f(\\cdot)$ in experiments relies solely on anchor vectors as classifier weights.

Question 2: Are the anchor vectors treated as learnable parameters? It would be helpful to clarify the background of LDL.

Yes, the anchor vectors are indeed treated as learnable parameters that play a fundamental role in the model's architecture. These vectors are optimized during minimizing the Kullback-Leibler (KL) divergence between the predicted label distributions and the ground-truth distributions, as formalized in Equation (3) of the paper. The learning process is further refined through the proposed Inter-Anchor Angular Regularization (IAR) term, which explicitly penalizes excessive cohesion between anchor vectors by minimizing their pairwise cosine similarities.

LDL (Label Distribution Learning) is an effective approach to accurately estimate the entire conditional distribution of labels according to a set of feature variables. This task receives increasing attention both in the field of statistics and machine learning, as the information about the entire distribution is crucial in scenarios that are sensitive to risk, extremes, or uncertainty, such as drug efficacy prediction or emotion recognition. Various kinds of techniques, such as model calibration or mixture density neural network can be utilized to estimate the entire conditional distribution using the training samples that are labeled only with the mean or mode of the underlying true conditional distribution, which are beneficial in the tasks where the true label distributions are unavailable. However, there remain a large number of real-world scenarios where the true distributions are readily available. To address this kind of scenarios, LDL is proposed to learn a multivariate regressor that maps a set of feature variables to the entire conditional distribution of labels according to a training set where each instance is labeled with a label distribution. Compared to the cases without true label distributions, LDL is capable of predicting the entire conditional distribution of labels more accurately, as it is directly supervised by the true label distributions.

Dear Reviewer 21xv,

Thank you for your positive assessment on our work. We sincerely appreciate the time and effort that you have dedicated to evaluating our work. We have carefully considered all the comments and have revised the paper accordingly. Below, we provide a point-by-point response to your suggestions.

Question 1: Can IAR be extended to architectures without explicit anchor vectors, such as attention-based or Transformer models?

Our current theoretical framework and methodology are specifically designed for models with explicit anchor vectors, as the core mechanism relies on regularizing the angular separation between these anchor vectors. Therefore, our method is not applicable to models that do not involve anchor vectors. As elaborated in Section 6 of the main text, our proposed inter-anchor angular regularization (IAR) term is not directly compatible with tree-based LDL algorithms since their learning process does not incorporate anchor vectors. On the other hand, our approach actually can be integrated with transformer or attention-based architectures, as the IAR term solely constrains the weight matrix of the neural network's output layer.

Question 2: How sensitive is the performance to the choice of entropy threshold in ECA? Would a learned threshold work better?

Actually, the entropy threshold does not participate in the model learning process; rather, it serves as a parameter in the model evaluation phase. The choice of this threshold parameter can indeed significantly impact the presented performance. For instance, if the threshold is set too small, the majority of test samples would be treated as high-entropy samples, thereby biasing the evaluation results toward high-entropy samples. To address this problem, our paper proposes treating the threshold as a distribution and computing the mean performance across all possible threshold values. In practice, a more reasonable approach is to determine the threshold based on specific task requirements—that is, defining what entropy level qualifies a sample as high-entropy or low-entropy according to the application's needs. As for whether a learnable threshold could be better, we argue that adaptive threshold would not be particularly beneficial, since the threshold is purely a parameter in model evaluation and does not participate in model training.

Dear Reviewer r9iZ,

Thank you for your positive assessment on our work. We sincerely appreciate the time and effort that you have dedicated to evaluating our work. We have carefully considered all the comments and have revised the paper accordingly. Below, we provide a point-by-point response to your suggestions.

Question: All baselines are non-deep or shallow deep models. Demonstrating IAR on a modern backbone (e.g., ViT-LDL) would strengthen the claim of universality.

We selected two datasets where instance features are encoded by raw image information to demonstrate the effectiveness of the proposed IAR term in deep learning models. We apply the proposed IAR term to the ViT-LDL algorithm while maintaining strict consistency with the experimental procedure utilized in the main text. The next two tables show the performance of the ViT-LDL model with IAR term and the ViT-LDL model without IAR term. It can be seen from the next two tables that our proposed IAR improves the performance of ViT-LDL in most cases.

Dear Reviewer 3R2w,

Thank you for your constructive comments on our paper. We sincerely appreciate the time and effort that you have dedicated to evaluating our work. We have carefully considered all the comments and have revised the paper accordingly. Below, we provide a point-by-point response to your suggestions.

Question 1: How about the performances of different algorithms on other LDL metrics (e.g., Chebyshev distance, Intersection similarity, etc.)?

Due to space limitations, we have only shown the performance of the algorithm using KL divergence and cosine similarity measures. Nevertheless, we have shown the performance of the algorithm using Chebyshev distance and intersection similarity in the table below. Due to the character limit for the rebuttal, we only present results under the ECA aggregation strategy for these two metrics, omitting detailed comparisons between low- and high-entropy subsets.

According to the results in the above table, it can be seen that under the Chebyshev distance and intersection similarity metrics, our algorithm achieves state-of-the-art performance on the Jaffe, BU-3DFE, Natural Scene, Emotion6, Abstract Painting, M2B, and Movie datasets. Notably, while it does not attain the best performance on the Music Mood dataset, it remains highly competitive, with performance very close to the top.

Question 2: On some high-entropy datasets, the performance gap between IAR and other algorithms is not significant, while on low-entropy datasets, IAR shows a noticeable performance gap compared to some SOTA algorithms. The authors should further analyze the underlying reasons for these phenomena.

The performance gap is more pronounced on low-entropy datasets than on high-entropy datasets because prediction errors exhibit greater variance in low-entropy datasets compared to high-entropy ones. We demonstrate this claim through data simulation with the following procedure:

1. First, we uniformly generate 10,000 label distributions $[\boldsymbol z_n]_{n=1}^{10,000}$ with varying entropy levels.
2. For each label distribution $\boldsymbol{z}_n$, we randomly generate 10,000 predicted label distributions and compute their KL divergence, cosine similarity, Chebyshev distance, and intersection similarity with respect to $\boldsymbol{z}_n$.
3. We then partition $[\boldsymbol z_n]_{n=1}^{10,000}$ into nine groups based on entropy intervals: [0, 0.2], (0.2, 0.4], (0.4, 0.6], (0.6, 0.8], (0.8, 1], (1, 1.2], (1.2, 1.4], (1.4, 1.6], (1.6, $+\infty$).
4. Finally, we compute the variance of all prediction performance metrics within each group.

The results (shown in the table below) reveal that label distributions with higher entropy exhibit smaller variance in prediction errors. This explains why the performance gap is more noticeable in low-entropy datasets than in high-entropy ones.

Question 3: The impact of the hyperparameter $\alpha$ on model performance varies across different datasets. The authors should further analyze the underlying reasons for the hyperparameter $\alpha$ on model performance varies across different datasets, and provide practical recommendations for its application.

We analyzed the underlying reasons for the hyperparameter $\alpha$ on model performance varies across different datasets, and provided practical recommendations for its application as follows.

Underlying reasons for the hyperparameter $\alpha$ on model performance varies across different datasets. The hyperparameter $\alpha$ balances the relative importance between the training error (KL divergence between the model-output label distribution and the true label distribution) and the IAR regularization. Since the converged training error (KL divergence value) is influenced by the number of labels, data distribution, and the noise level of the dataset, the optimal weight ($\alpha$) for the IAR term should be dataset-dependent. Furthermore, variations in data distribution across datasets lead to different degrees of over-uniformity when the training error is converged, consequently requiring different levels of IAR regularization during training. This further necessitates dataset-specific $\alpha$ values.

Recommended settings. As analyzed in lines 249-254 of our paper, we recommend $\alpha=40$ as a default setting for most datasets, as it approaches optimal performance across diverse scenarios. For further performance optimization, we suggest the following empirical guidelines:

1. When the training set contains predominantly high-entropy label distributions, consider using $\alpha<40$.
2. For datasets with predominantly low-entropy label distributions, larger $\alpha$ values are preferable.

Thanks for the responses. I appreciate the further results on more evaluation metrics and the analysis of the impact of the hyperparameter $\alpha$. However, for Q2, although the authors give a simulation-based explanation of how prediction error variance differs across entropy levels, I believe the paper would benefit from the analysis of actual data or a theoretical perspective to further study the phenomenon.

We sincerely appreciate the reviewer's insightful comments regarding the need for further validation on real-world data and theoretical analysis. In response, we have conducted additional experiments and analyses as follows:

​​Real-world Data Validation: ​​We have verified this phenomenon on real-world datasets by replacing the artificially generated label distributions in our simulation experiments with actual dataset label distributions. To facilitate comparison across datasets with varying dimensions of label distributions, we present normalized entropy values rather than raw entropy values in the following table (as different datasets have different dimensionality in their label distributions, making raw entropy values incomparable). The experimental results on real-world data consistently demonstrate that high-entropy datasets (Jaffe, BU-3DFE, Movie, Music Mood) exhibit significantly lower variance in prediction performance compared to low-entropy datasets (Natural Scene, Emotion6, Art Painting, M2B).

We also note an interesting observation within the low-entropy dataset group: while Emotion6 and Art Painting show similar entropy levels to other low-entropy datasets, their KL divergence variance is substantially higher. We believe that this occurs because the KL metric is particularly sensitive to zero elements, and these two datasets contain abundant zero elements in their label distributions.

Theoretical Validation: Given a true label distribution $[d_1,d_2,\dots,d_n]$, where $d_1<d_2<\cdots<d_n$ can be assumed without loss of generality, the squared Euclidean distance between any predicted label distribution $[z_1,z_2,\dots,z_n]$ and the true label distribution is $y=(d_1-z_1)^2+(d_2-z_2)^2+\cdots+(d_n-z_n)^2$, which can be utilized to quantify the performance the label distribution predictor. The minimum value of $y$ is zero when $z_i=d_i$ holds for $i=1,2,\dots,n$. The maximum value of $y$ is $(1-d_1)^2+d_2^2+\cdots+d_n^2$ when $z_1=1$, $z_2=z_3=\cdots=z_n=0$. Then the range of the performance is $r=(1-d_1)^2+d_2^2+\cdots+d_n^2$. Now, let $\boldsymbol d = [d_1,d_2,\dots,d_n]$ and $\boldsymbol h = [h_1,h_2,\dots,h_n]$ denote two true label distributions, where the Gini coefficient of $\boldsymbol d$ is smaller than the Gini coefficient of $\boldsymbol h$, i.e., $(1-h_1^2-\cdots-h_n^2) - (1-d_1^2-\cdots-d_n^2) = c > 0$, and $h_1+c/2>d_1$. It should be noted that we use Gini coefficient here to quantify the uncertainty of a label distribution, which is similar to entropy yet easier to calculate than entropy. According to $(1-h_1^2-\cdots-h_n^2) - (1-d_1^2-\cdots-d_n^2) = c > 0$, we have:

$(1-h_1^2-\cdots-h_n^2) - (1-d_1^2-\cdots-d_n^2) = c > 0$

$\Longleftrightarrow d_1^2+d_2^2+\cdots+d_n^2 = h_1^2+h_2^2+\cdots+h_n^2 + c > h_1^2+h_2^2+\cdots+h_n^2 + 2(d_1 - h_1)$

$\Longleftrightarrow d_1^2+d_2^2+\cdots+d_n^2 - 2d_1+1 > h_1^2+h_2^2+\cdots+h_n^2 - 2h_1+1$

$\Longleftrightarrow (1-d_1)^2+d_2^2+\cdots+d_n^2 > (1-h_1)^2+h_2^2+\cdots+h_n^2$

Therefore, the performance range on a true label distribution with lower uncertainty is broader than the performance range on a true label distribution with higher uncertainty if $h_1+c/2>d_1$ (this requirement can be satisfied in most practical cases (about 90% cases in the real-world datasets in our paper)).

We believe these additional experiments and theoretical analyses substantially strengthen our original findings and provide more comprehensive evidence for the observed phenomenon. Please don’t hesitate to let us know if additional details or revisions would be helpful if there are any remaining points that require further clarification.