Predicting Label Distribution from Ternary Labels

Responses to Weaknesses

Thank you for your suggestions. Under the space restriction, we will incorporate some background knowledge about three-way philosophies in the revised version. Besides, we will rearrange the Figure 2 and Figure 3, and correct the mathematical formula in Figure 3.

Responses to Questions

Thank you for your questions! Take GLLE as an exmple, the objective function of GLLE is

where$f(\\boldsymbol x\_n; \\boldsymbol W)$ is a predictive model with learnable parameters of $\\boldsymbol W$, $\\boldsymbol y\_n$ is a logical label vector, $\\Omega(\\boldsymbol W)$ is the regularization term. If we apply CateMO for GLLE, the objective function will be

where $\\boldsymbol s\_n$​ is a ternary label vector.

Besides, in terms of how to set the lambda parameters, we provide the following two methods:

• End-to-end Learning. The most straightforward way to determine the parameters $\\underline{\\lambda},\\lambda,\\overline{\\lambda}$ is to co-optimize these parameters with other parameters in the learner in an end-to-end manner. This approach is capable of obtaining parameters adapted to the characteristics of the data distribution, but corresponds to a complex constrained optimization problem that requires a tailored optimization algorithm since the constrains of $\\underline{\\lambda},\\lambda,\\overline{\\lambda}$ are interdependent.

• Pre-annotations Fitting. The parameters can also be estimated from a small number of pre-annotated data pairs $\\{(s\_n,z\_n)\\}\_{n=1}^L$, since $p(s|z)$ possesses a relatively stable morphology across different datasets. Figure 2 in the submitted PDF in global responses shows the distribution of some of our annotation results on the Painting, Music and JAFFE datasets. Specifically, we randomly choose an instance and a label from the JAFFE, Painting, and Music datasets, and asked the experts to simultaneously annotate the instance-label relationship by a ternary value and a description degree value. The above process is repeated six hundred times (two hundred times for each dataset). All annotation results are collected and recorded as $\\mathcal A=\\{(s\_n,z\_n)\\}\_{n=1}^L$. Figure 2 in the submitted PDF in global responses shows the empirical distribution of $p(s|z)$ according to $\\mathcal A$, which indicates that there are no significant differences in the empirical distributions of $p(s|z)$ across datasets. Therefore, we can estimate the parameters of CateMO reliably with a small amount of pre-annotated data pairs. Formally, we obtain the parameters by maximum likelihood estimation, i.e.,

Besides, in order to provide users with a more intuitive understanding of CateMO's parameters, we show the shape of CateMO with varying $\\underline{\\lambda},\\lambda,\\overline{\\lambda}$ in Figure 3 in the submitted PDF in global responses, which visualizes how the parameters $\\underline{\\lambda},\\lambda,\\overline{\\lambda}$ detemine CateMO.

Responses to Weakness

We really appreciate your valuable comments. Essentially, partial multi-labels, weakly-supervised multi-labels, semi-supervised multi-labels, and multi-labels with noise are essentially weaker versions of binary labels (i.e., multi-labels) because they either contain noise (i.e., incorrectly supervision data) or miss the correct supervision data. In contrast, ternary labels represent an enhanced version of binary labels, as ternary labels not only provide definitely relevant labels and definitely irrelevant labels but also leverage the option of uncertain label to identify the labels that are more prone to generating noise. Besides, the detailed illustration about the relationship between this paper and weakly-supervised multi-label learning is shown in Responses to Weakness (1) for Reviewer 3ktH. Under the space restriction, we will incorporate a discussion about the relationships between our paper and other similar paradigms in the revised version.

Responses to Questions

Thank you for your insightful questions! In terms of the cases where the number of uncertain labels is excessively large, we give the following proposition:

Proposition: Given an uncertain label, i.e., $p(s=0)=1$, we denote the expected approximation errors produced by annotating the label with a ternary value and a binary value as $\\psi\_s$ and $\\psi\_b$, respectively. Suppose that $\\hat\\xi\\sim \\rm{Uni}(\\hat \\xi\\mid \\hat\\tau \\le \\hat \\xi \\le \\hat\\kappa)$, $\\rho\\sim \\rm{Uni}(\\rho\\mid 0\\le\\rho\\le 1)$, $[\\tau,\\kappa]\\sim \\rm{Uni}([\\tau,\\kappa]\\mid 0\\le \\tau\\le \\kappa\\le 1)$, we have

Furthermore, if $[\\hat\\tau,\\hat\\kappa]\\sim \\rm{Uni}([\\hat\\tau,\\hat\\kappa]\\mid 0\\le \\hat\\tau\\le \\hat\\kappa\\le 1)$, we have

The above proposition reveals the advantage of ternary labels over binary labels when all labels are uncertain labels, i.e., $p(s=0)=1$. According to the above proposition, the expected approximation error of binary annotating exceeds that of ternary annotating when $\\frac29(\\hat\\tau^2+\\hat\\kappa^2+\\hat\\tau\\hat\\kappa) - \\frac13(\\hat\\tau+\\hat\\kappa)+\\frac{1}{12}\\le 0$. Furthermore, if we equiprobably set $\\hat\\tau$ and $\\hat\\kappa$ to any two values in the interval $0\\le \\hat\\tau\\le \\hat\\kappa\\le 1$, then the probability that the ternary annotating possesses lower expected approximation error is 91% approximately. Besides, in terms of the comparative experiments, we will clarify the fairness of each comparative experiments in greater detail under the space restriction.

Responses to Weakness (1) and Question (2)

We are very grateful for your precious suggestions. We additionally perform experiments to analyze the parameter sensitivity. The results are shown in Figure 1 in the submitted PDF in global responses, which will be added to the revised version if space permits.

Actually, there are only three parameters in CateMO, since the paremeter $\\hat z$ can be determined by the other parameters according to Eq. (12). In terms of how to determine the parameters $\\underline{\\lambda},\\lambda,\\overline{\\lambda}$ in Eq. (10), we offer the following two approaches.

• End-to-end Learning. The most straightforward way to determine the parameters $\\underline{\\lambda},\\lambda,\\overline{\\lambda}$ is to co-optimize these parameters with other parameters in the learner in an end-to-end manner. This approach is capable of obtaining parameters adapted to the characteristics of the data distribution, but corresponds to a complex constrained optimization problem that requires a tailored optimization algorithm since the constrains of $\\underline{\\lambda},\\lambda,\\overline{\\lambda}$ are interdependent.

• Pre-annotations Fitting. The parameters can also be estimated from a small number of pre-annotated data pairs $\\{(s\_n,z\_n)\\}\_{n=1}^L$, since $p(s|z)$ possesses a relatively stable morphology across different datasets. Figure 2 in the submitted PDF in global responses shows the distribution of some of our annotation results on the Painting, Music and JAFFE datasets. Specifically, we randomly choose an instance and a label from the JAFFE, Painting, and Music datasets, and asked the experts to simultaneously annotate the instance-label relationship by a ternary value and a description degree value. The above process is repeated six hundred times (two hundred times for each dataset). All annotation results are collected and recorded as $\\mathcal A=\\{(s\_n,z\_n)\\}\_{n=1}^L$. Figure 2 in the submitted PDF in global responses shows the empirical distribution of $p(s|z)$ according to $\\mathcal A$, which indicates that there are no significant differences in the empirical distributions of $p(s|z)$ across datasets. Therefore, we can estimate the parameters of CateMO reliably with a small amount of pre-annotated data pairs. Formally, we obtain the parameters by maximum likelihood estimation, i.e.,

Besides, in order to provide users with a more intuitive understanding of CateMO's parameters, we show the shape of CateMO with varying $\\underline{\\lambda},\\lambda,\\overline{\\lambda}$ in Figure 3 in the submitted PDF in global responses, which visualizes how the parameters $\\underline{\\lambda},\\lambda,\\overline{\\lambda}$ detemine CateMO.

Responses to Weakness (2) and Question (3)

Thank you for your questions. In Figure 3 in the submitted PDF in global responses, we show, across three datasets, the proximity of the label distributions recovered by CateMO to the ground-truth label distributions. From the experimental results, it can be seen that the label enhancement algorithm based on CateMO achieves significant advantages over others.

Responses to Question (1)

The proposed ternary generation function can be expressed as

where $\\mathrm{softmax}(a,b,c)=[e^a/(e^a+e^b+e^c),e^b/(e^a+e^b+e^c),e^c/(e^a+e^b+e^c)]$. It can be seen that the proposed ternary generation function is essentially a softmax function. The reason why we use the softmax function as the basic form is that the softmax function is most commonly used for modelling probability mass functions of discrete variables and offers great simplicity in differential calculus.

The motivation for the components $-z^2,-(z-\\hat z)^2,-(z-1)^2$ is that the ternary generation functions are expected to maintain the following properties:

• $s$​ is more likely to be $-1$​ when $z$​ is close to $0$​.
• $s$ is more likely to be $1$ when $z$ is close to $1$.
• $s$ is more likely to be $0$ when $z$ is far away from both $0$ and $1$​.

Finally, the parameters $\\underline \\lambda,\\lambda,\\overline \\lambda$ are used to control the strength of each component in the softmax function, similar to the precision parameter in a Gaussian distribution.

Responses to Weakness (1)

(1) Responses to the concern about "Weakly Supervised Multi-Label Learning via Label Enhancement".

We greatly appreciate the constructive feedback you have provided on our paper. It should be highly emphasised that the ternary labels in our paper and the WSML (i.e., weakly-supervised multi-labels) are inherently different despite sharing the same representation ($-1$, $0$, $1$). Specifically, they differ in the following aspects.

Admittedly, we acknowledge that our paper may not illustrate this aspect thoroughly. We pledge to improve this section in accordance with your valuable suggestions.

(2) Responses to why there is no LE (i.e., Label Enhancement) method for ternary labels.

As illustrated in the above table, WSML and ternary labels differ in the origin of the label with value of $0$​. Therefore, we believe that it is unjustified to apply the algorithms of WSML directly for ternary labels. Nonetheless, we acknowledge your suggestions and we will further improve the claims.

Responses to Weakness (2)

(1) Responses to the suggestion that some existing LE methods should be compared, e.g., [a], [b], [17].

Our proposed CateMO distribution, serving as a plug-and-play component that enables most standard binary LE algorithms to be applied to ternary labels, rather than an algorithm that outperforms existing binary LE algorithms. Therefore, our proposed CateMO is not comparable to the papers [a], [b], [17]. Nevertheless, we will discuss the relationships between the papers [a], [b], [17] and our paper appropriately since these papers are related to our work.

(2) Responses to the concern about datasets.

The datasets used for the experiments in the paper [17] are not applicable to the research context of this paper because those datasets lack ternary labels. However, ternary labels cannot be obtained by setting the binary labels to uncertain labels directly since the uncertain labels must be the labels that are really difficult to determine whether they are relevant to the instance.

Responses to Question (1)

The fundamental differences between WSML and ternary labels have already been illustrated in "Responses to Weakness (1)". We will comprehensively disscuss their relationships in the revision.

Responses to Question (2)

Thank you for your suggestions. Our paper primarily concentrates on the quantitative and qualitative relationships between ternary labels and label description degrees. Unfortunately, due to page limitation, possible extensions of our proposal were not elaborated upon. Under the space restriction, we will endeavor to incorporate a discussion of these extensions in the revised version.

Responses to Question (3)

In our paper, we have not proposed an "algorithm". As illustrated in Responses to Weakness (2), our proposed CateMO is a probability distribution, a plugin that is able to work with most discriminative/generative LE algorithms. Therefore, it is unnecessary to discuss the advantages/disadvantages of our proposed CateMO compared to the existing discriminative/generative LE algorithms.

Responses to Question (4)

There is no objective function to show in our paper since we have not proposed a “learning algorithm” in our paper. The mathematical details of our proposed CateMO are shown in Eq. (12), and Eq. (13) shows how CateMO works in conjunction with existing LE algorithms. Besides, in terms of how to enrich the supervision information, various LE algorithms have their own logic to enrich the supervision or to generate the label description degrees, e.g., GLLE uses the smoothness assumption, and LELR uses the potential ranking relationships within binary labels. Our proposed CateMO merely models the probabilistic relationship between the label description degree and the ternary label, which is theoretically guaranteed to satisfy the proposed basic assumptions.

Responses to Question (5)

The results of statistical significance test have been shown in the Table 2 of Appendix A.4 of our paper.

Responses to Question (6)

Thank you for your suggestion. We additionally perform experiments to analyze the parameter sensitivity. The results are shown in Figure 1 in the PDF of global response, which will be added to the revised version as space permits. More detailed discussion about the parameters $\\underline \\lambda,\\lambda,\\overline \\lambda$ can be found in the "Responses to Weakness (1) and Question (2)" section for Reviewer uVMp.

Dear Reviewer:

We would like to kindly inquire whether our responses have adequately addressed the concerns you raised during the review process. Besides, we are also seeking to ascertain if there are any new question that you would like us to clarify.