Divide and Conquer: Learning Label Distribution with Subtasks

Many thanks for your precious comments! We have provided point-by-point responses to your questions below.

Comment 1: Why do the authors define the diversity as in Definition 4.3, instead of the cosine similarity used in Equation 1?

Response: In Definition 4.3, we use $\bar{\boldsymbol{d}}$ to assign weights to each subtask pair, which can make the two subtask label spaces with different high-frequency labels considered more differentiated. This helps evaluate diverse subtask label distributions. In Equation (1), we use cosine similarity instead for optimization purposes, since element-wise XOR operator is a discrete, non-continuous operation. We will clarify this distinction in the manuscript to improve readability.

Comment 2: Some steps in Theorem 4.4 could be made more clear. For example, the proof relies on the assumption that $q$ is a normalization constant, but does not discuss what happens if this is not the case. Moreover, the steps following Equation 8 are not clear: what does it mean that $\sum [p(d)]_{j}$ is "given"? Furthermore, there is implicitly a linearity assumption made to deduce that $p(v) = v$, hence more details on the derivation would be helpful.

Response: Below, we provide a detailed explanation of the points raised:

• If $[q(\cdot)]_j$ is not a constant for all $j$, each element has its own scaling factor, which is atypical and it is difficult to output a probability simplex.
• What we mean is that the term $\sum_{j=1}^{L} [p(\boldsymbol{d})]_{j}$ must be known a priori when solving for $d_k$ in Equation (8).
• The choice $[p(\boldsymbol{v})]_j = v_j$ for all $j$ is the simplest nontrivial solution consistent with the problem constraints. We will explicitly call out the linearity assumption and justifies why it’s necessary (to avoid cross-dependencies and preserve normalization).

Comment 3: The writing of the paper could be improved, for example the notation used is not always properly introduced (cf. Algorithm 1 where $j$ appears without proper definition, making it hard to understand the algorithm).

Response: In this context, $j$ denotes the index of the label space. While we initially used a more compact pseudocode style for brevity, we acknowledge that this could lead to ambiguity. We will revise the algorithm to provide a more detailed and unambiguous formulation in the updated version of the paper.

Specifically, Line 4 in Algorithm 1 can be expanded into:

$\mathcal{Y}^{(t)} \leftarrow \lbrace \varnothing \rbrace$;

for $j=1$ to $L$ do

    if $M_{tj}=1$ then

         $\mathcal{Y}^{(t)} \leftarrow \mathcal{Y}^{(t)} \cap \lbrace y_j \rbrace$;

    end if

end for

Line 9 in Algorithm 1 can be expanded into:

$\boldsymbol{D}^{(t)} \leftarrow []$;

for $j=1$ to $L$ do

    if $y_j \in \mathcal{Y}^{(t)}$ then

        $\boldsymbol{D}^{(t)} \leftarrow [ \boldsymbol{D}^{(t)} | \text{clip}(\boldsymbol{d}_{\bullet j}, \varepsilon, 1) ]$;

    end if

end for

Comment 4: Furthermore, there is no clear intuition or theoretical explanation for why subtasks should help with the LDL primary task, and why diversity and information should be maximized when designing good subtasks.

Response: First, the reason why subtasks help with the primary task can be explained from the perspective of shared representation learning, which forces the model to learn more general and discriminative feature representations. These features may capture patterns that are beneficial to the primary task. Moreover, empirically speaking, simple techniques that are similar to our method (e.g., label powerset, dummy variable, label smoothing, etc.) have been demonstrated to achieve performance improvements. Second, the reason why diversity and information should be maximized is to ensure that each subtask contributes unique knowledge, avoiding trivial or overlapping learning signals. Also, diverse and informative subtasks can act as implicit regularization, preventing the model from relying too heavily on spurious label correlations in the primary task.

Many thanks for your precious comments! Responses to your concerns are as follows.

Comment 1: There are limited performance improvements in some cases, particularly with the Yeast_ series datasets.

Response: For LDL, even small changes in metrics can indicate significant performance improvements. This is because the values involved in the calculation are constrained by probability simplex, which limits the range of results and leads to small divergence. For example, on the $\mathtt{Yeast\\_diau}$ dataset, the K-L divergence between the ground truth and a uniform label distribution matrix (values are all $\frac{1}{L}$, where $L$ is the number of labels) is 0.0158. This can be considered as the worst performance. DF-LDL, a strong baseline, achieves 0.0131. To ensure the reliability of the experimental results, we repeat the experiments and provide the standard deviation of the results, and perform significance tests (pairwise t-test at 0.05 significance level) to confirm that the observed changes are statistically significant.

Comment 2: There are some non-intuitive parts of the analysis. The notation $[\cdot]$ is used to denote the set of natural numbers, but $[\cdot]_i$ is used to represent the element at the $i$-th position of a vector, which could be confusing for readers sometimes.

Response: We appreciate this observation. To avoid ambiguity, we will revise the notation by using parentheses for indexing (i.e., $(\cdot)_i$ for the $i$-th element of a vector) and reserve square brackets exclusively for set definitions.

Comment 3: Regarding Section 4.2, what benefits does the reconstructability of subtask label distribution bring?

Response: The purpose of Section 4.2 is to demonstrate that, under certain conditions, the subtask label distributions can indeed reconstruct the primary task label distribution. In Section 5, we concatenate the subtask label distribution with the representation to serve as input to a label distribution estimator, simulating this reconstruction process. In essence, reconstructability guarantees that subtasks are not just arbitrary auxiliary objectives but are structurally aligned with the primary LDL problem. Therefore, it is crucial for the overall validity of the $\mathcal{S}$-LDL framework to prove that the subtask label distributions can yield the original distribution.

Comment 4: Regarding Theorem 4.4, why there are three conditions? More explanations would be helpful.

Response: Below, we provide explanation of the conditions:

• $\mathcal{G}$ is connected: This ensures that the proportions between all description degrees of the primary label distribution are known. If $\mathcal{G}$ is not connected, the proportions of description degrees corresponding to labels on different connected components would remain unknown.
• $\mathcal{G}$ covers all labels in the label space: The union of the subtask label spaces must encompass the entire primary label space. If any labels are not included in any subtask label space, the description degrees for those labels would be unknown. Hence, this is a necessary condition.
• The corresponding description degrees of all cut vertices of $\mathcal{G}$ are not zero: When there are cut vertices in $\mathcal{G}$, the non-zero description degrees corresponding to all the cut vertices are necessary to keep the proportions known.

Many thanks for your precious comments! We have provided point-by-point responses to your questions below.

Comment 1: It is better to provide some evidence that existing methods cannot deal with the contradiction.

Response: In Section 1 & 2, we have provided such evidence. On the one hand, we highlight how existing methods relying on prior/expert knowledge (e.g., [Chen et al., 2020; Wu et al., 2019; Yang et al., 2017a]) suffer from poor generalization, as they cannot adapt to new domains. On the other hand, we demonstrate that conventional LDL techniques use solely the primary task distribution (e.g., [Jia et al., 2019; 2023; Ren et al., 2019; Wen et al., 2023]) as supervised data. These comparisons directly show why existing solutions cannot resolve the contradiction, and the essential reason behind this is highly related to the design logic of these methods.

Comment 2: The claim of "furnish additional supervised data" seems not rigorous and might cause misunderstanding, since you have not involved more external data. Maybe it is better to use "supervision" or "supervised information".

Response: The claim is indeed not meant to imply external data but rather newly constructed data for supervision. We will revise the wording to clarify this in the manuscript.

Comment 3: Some related works in MLL with partitioning of the label space provided theoretical justification. However, there is no theoretical support for the proposed method.

Response: We sincerely appreciate the reviewer's insightful comment regarding theoretical justification. While prior MLL works with label space partitioning do offer theoretical analyses, our method provides distinctive theoretical support through the following aspects. In Section 4.1, we have discussed the validity of subtask construction, ensuring the subtasks remain meaningful for the primary LDL objective. In Section 4.2, we prove that the primary label distribution can be theoretically reconstructed from subtask distributions, demonstrating information preservation. Additionally, in Section 4.3, we have provided a detailed analysis of the time complexity of subtask construction, establishing its scalability and efficiency. While our theoretical framework differs from related works in MLL with partitioning of the label space, these three pillars collectively justify the rationality of our design.

Comment 4: The performance gains on some datasets are marginal (less than 0.01). Is there any explanation?

Response: For LDL, even small changes in metrics can indicate significant performance improvements. This is because the values involved in the calculation are constrained by probability simplex, which limits the range of results and leads to small divergence. For example, on the $\mathtt{Yeast\\_diau}$ dataset, the K-L divergence between the ground truth and a uniform label distribution matrix (values are all $\frac{1}{L}$, where $L$ is the number of labels) is 0.0158. This can be considered as the worst performance. DF-LDL, a strong baseline, achieves 0.0131. To ensure the reliability of the experimental results, we repeat the experiments and provide the standard deviation of the results, and perform significance tests (pairwise t-test at 0.05 significance level) to confirm that the observed changes are statistically significant.

Comment 5: It seems that the references are a bit out-of-date. Most references are from or earlier than 2023. Is there more recent related work?

Response: We appreciate the reviewer's suggestion. While there are indeed some recent works in the field of LDL [1, 2], they currently lack official open-source implementations, making their effectiveness and reproducibility unverified. For reliability and fairness in comparison, we chose to focus on well-established methods with publicly available implementations.

[1] Label Distribution Learning Based on Horizontal and Vertical Mining of Label Correlations. TBD, 2024.

[2] Exploiting Multi-Label Correlation in Label Distribution Learning. IJCAI, 2024.

Many thanks for your comments! Responses to your questions are as follows.

Comment 1: How does it (LDL model) differ from a typical classification network?

Response: Although typical classification networks can formally produce outputs similar to label distribution learning (LDL) through Softmax, they differ significantly in essence, specifically manifested in the following aspects:

• Learning Objectives: Typical classification networks with Softmax outputs aim to learn a single-peaked probability distribution, emphasizing the certainty that a sample belongs to a specific category (i.e., the "one true label"). Their limitation lies in the inability to express the varying degrees of association between a sample and multiple labels. In contrast, LDL seeks to learn a multi-modal (possibly single-peaked or multi-peaked) label distribution, describing the degree of association between a sample and all labels. It is primarily suited for scenarios where semantic overlap or granularity differences exist among labels. The core challenge of LDL never lies in ensuring outputs satisfy a probability simplex, but rather in precisely fitting the description degrees.

• Semantic Interpretation: The Softmax output ensures that probabilities sum to 1, but only the highest probability label is meaningful (winner-takes-all). The label distribution output, however, determines probability distribution based on the relative descriptive strength among labels, allowing all labels to retain interpretability simultaneously.

• Underlying Philosophy: The classification model is decision-oriented, prioritizing clear classification boundaries. LDL model is description-oriented, pursuing fine-grained quantification of label associations.

Comment 2: What form does the training set take? Does each sample come with a soft label?

Response: In short, each sample $\boldsymbol{x}$ come with a label distribution $\boldsymbol{d}$, which is not a soft label. Soft labels are often introduced to improve the generalization of classification tasks, and they are still only related to a specific dominant class. Label distributions model ambiguity, where diversity is a property of the data itself. Therefore, label distributions may inherently follow a continuous or multi-peaked distribution (e.g., subjective human annotations). We have some rigorous statements about LDL in the problem definition in Section 3.1. These do not conflict with your observations based on ERM.

To clarify, our work follows a pure LDL paradigm, which is fundamentally distinct from traditional classification tasks. We believe this clarification adequately addresses the reviewer's concerns. We sincerely hope these issues will not negatively impact the overall rating of our contribution.