Towards Escaping from Class Dependency Modeling for Multi-Dimensional Classification

We want to express our sincere gratitude for your invaluable comments and suggestions. According to your comments in Claims And Evidence, Experimental Designs Or Analyses and Other Strengths And Weaknesses, we summarize the following questions. The point-to-point responses are given as follows:

-Q1: The assumption of Theorem 3.3 may not be easily satisfied in practice. Can the authors provide examples to demonstrate how it can be held?

Answer for Q1: We sincerely appreciate your insightful question regarding the practical validity of the assumption of Theorem 3.3.

The basic condition of Theorem 3.3 is that the joint probability $p_{Y}(y_{\diamond 1},y_{\diamond 2},\dots,y_{\diamond q}) \to 0$, which can guide selecting high-frequency class combinations when constructing $\mathcal{D}'$ in Eq.(10) (please refer to the answer for Q2 for Reviewer Kvke). Take dataset Flare1 as an example, the 323 examples present 14 distinct class combinations, where only 3 class combinations correspond to over 10 examples. In other words, the probability of most class combinations is less than 0.03.

As for the low-frequency class combinations involved in Theorem 3.3, we have $p^{\mathrm{(jt)}}(y_{\diamond},x_{\diamond},z_{\diamond})\le \int_{x_{\diamond}}\int_{z_{\diamond}}p^{\mathrm{(jt)}}(y_{\diamond},x_{\diamond},z_{\diamond})dz_{\diamond}\le p_{Y}(y_{\diamond 1},\dots,y_{\diamond q})\to 0,$ Accordingly, $p^{\mathrm{(jt)}}-\delta$ also tends to 0, which makes the Assumption 3.4 easily satisfied.

-Q2: The authors claim that the learned latent factor can capture "critical feature information". However, only empirical results are provided and there are no theoretical results or visualizations for justifying its effectiveness.

Answer for Q2: Thanks to the comment. Considering that most MDC data sets are tabular data sets, we have not provided visualizations of latent vectors before. For image data sets, we implement pretrained ResNet-18 as the feature extractor and then conduct the proposed approach. So the obtained latent vectors are still tabular.

Figure 1-3 at anonymous link https://anonymous.4open.science/r/DCOM-C8FE/ICML25_authors_response.pdf present the t-SNE visualizations for the first fold of dataset Voice, TIC2000 and Flickr, which show latent factors capture more compact manifold representations than the original features. We will incorporate these additions in the next version.

This dilemma urges us to develop CNN-based MDC approaches to obtain feasible feature maps directly in the future. Thanks again for this insightful comment.

-Q3: The authors claim that DCOM is computationally superior to MDC methods based on learning a graphical model. However, no experiments are performed to support this.

Answer for Q3: Thanks to the comment. In Section 2 (Related Work), we empirically mention the limitations of existing graphical MDC approaches which are widely recognized in previous works [1][2]. However, computational efficiency is neither the primary focus nor the core contribution of this paper. For comparision with the latest MDC approach PIST [3], which shares similar deep learning architecture, we list running times (500 epochs for training) on some data sets as follows which can show our efficiency:

-Q4: The proposed method lacks significant technical novelty.

Answer for Q4: Thanks to the comment. We would like to provide a more comprehensive response addressing this important concern. Existing MDC approaches mainly consider modeling class dependencies, where the hardness stems from the typical intercoupling within multiple dimensions. To address this issue, this paper proposes a dependency-free framework and establishes a novel technical route to solve MDC problems. As shown in our experimental results, the proposed approach which only employs conventional MLPs has yielded statistically significant performance improvements over state-of-the-art dependency-modeling baselines.

-Q5: The authors did not compare DCOM to MDC methods built upon PGMs. The claim that DCOM is more computationally efficient is not supported by experimental evidence.

Answer for Q5: Please refer to the answer for Q3.

-Q6: The learned latent variable lacks interpretability.

Answer for Q6: Please refer to the answer for Q2.

---

[1] B.-B. Jia, M.-L. Zhang. Decomposition-based classifier chains for multi-dimensional classification. IEEE TAI, 2022, 3(2): 176-191

[2] M. Zhu, S. Liu, and J. Jiang, “A hybrid method for learning multidimensional Bayesian network classifiers based on an optimization model,” Applied Intelligence, vol. 44, no. 1, pp. 123–148, 2016.

[3] Huang, T., Jia, B.-B., and Zhang,M.-L. Deep multi-dimensional classification with pairwise dimension-specific features. IJCAI'24, pp. 4183–4191.

We want to express our sincere gratitude for your invaluable comments and suggestions. The point-to-point responses are given as follows:

-Q1: I couldn't find details of the encoding network. If I missed some part of the paper, could you give me a pointer? Otherwise, could you add it in the next version?

Answer for Q1: Thanks to the comments. Please kindly refer to ​the right column of page 7 (lines 375–384) in Section 4.1.4 (IMPLEMENTATION DETAILS), the encoding network $\mathcal{G}$ is a multi-layer perceptron (MLP) with one hidden layer, configured with hidden dimension of 512. The output dimension, i.e., the dimensionality of latent variable $Z$ is also set as 512. All activation functions are fixed as ReLU followed by a dropout layer with dropping probability of 0.5. We truly hope this clarification can help elucidate details of the encoding network and we will ensure that the implementation details are more prominently highlighted in future revisions.

-Q2: Could you make the relations between $\log p_{X|Z}(\boldsymbol{x}_{i}|\boldsymbol{z}_i)$ in equation (13) and $\boldsymbol{z}_i$ clearer?

Answer for Q2: We sincerely apologize for the lack of clarity and thank you for identifying this ambiguity. Below we provide a detailed clarification of the relationship between $\log p_{X|Z}(\boldsymbol{x}_i|\boldsymbol{z}_i)$ in Eq.(13) and the latent variable $\boldsymbol{z}_i$:

The term $\log p_{X|Z}(\boldsymbol{x}_i|\boldsymbol{z}_i)$ represents the logarithmic posterior probability of reconstructing input $\boldsymbol{x}_i$ given the latent variable $\boldsymbol{z}_i$, where $\boldsymbol{z}_i$ serves as the input to reconstruction network $\mathcal{R}$. The reconstruction network $\mathcal{R}$ will then output the mean vector $\boldsymbol{\mu}_i$ and standard deviation vector $\boldsymbol{\sigma}_i$, i.e., key parameters of the assumed multivariate Gaussian $\mathcal{N}(\boldsymbol{\mu}_i,\boldsymbol{\sigma}_i^2 \mathbf{I})$.

-Q3: How do $\mu_{ia}$ and $\delta_{ia}$ related to $\boldsymbol{z}_i$?

Answer for Q3: Please kindly refer to the answer for Q2.

Considering that there is no $\delta_{ia}$ in Eq.(13), we suppose $\delta_{ia}$ in your question is $\sigma_{ia}$. $\boldsymbol{z}_i$ serves as the input to reconstruction network $\mathcal{R}$ and $\mathcal{R}$ will output the mean vector $\boldsymbol{\mu}_i$ and standard deviation vector $\boldsymbol{\sigma}_i$.

$\mu_{ia}$ and $\sigma_{ia}$ are the $a$-th element of $\boldsymbol{\mu}_i$ and $\boldsymbol{\sigma}_i$ respectively. So they are related directly through the reconstruction network $\mathcal{R}$ as the input and output vectors.

Besides, Eq.(13) aims at inducing the reconstruction loss, i.e., Eq.(14). Thus in the practical algorithm, the reconstruction network $\mathcal{R}$ only needs to output mean parameters $\boldsymbol{\mu}_i$.

-Q4: Which kind of reconstruction network has been used in the experiments?

Answer for Q4: Please kindly refer to the answer for Q1. All implementation details are presented in ​the right column of page 7 (lines 375–384) in Section 4.1.4. All neural networks ($\mathcal{G}, \mathcal{R}$, $\mathcal{T}$ and {$\mathcal{H}_j|1\le j\le q$}) are employed as the multi-layer perceptron (MLP) with one hidden layer, configured with hidden dimension of 512 (except for the feature extractor for image data sets).

-Q5: Have you assessed the robustness of DCOM under the presence of either noisy features or small data sets? Please all refer to "Claims And Evidence" for my detailed comments on this point.

Answer for Q5: Thank you for raising this critical question about the robustness of DCOM under noisy features or limited size of data sets. Actually, considering the conditional probability $p(\mathcal{G}(\boldsymbol{x})|\boldsymbol{x})$ is inherently describing a deterministic event, we introduce a minor perturbation on $\boldsymbol{x}$ before inputting it into the encoding network $\mathcal{G}$ (please kindly refer to Eq.(4)). In other words, we have added random noise to the original MDC data sets as Denoising Auto-Encoders [1]. We will make this clearer in the revised version.

As for the small data sets, We have rigorously evaluated DCOM across datasets with varying sizes to assess its generalization capability (please kindly refer to Table 5 in Appendix B). Small data sets such as Flare1, Oes97 and Jura only involve less than 400 samples. Though these results demonstrate DCOM's ability to handle limited data, we acknowledge (as correctly noted by you) that the performance advantage is more pronounced on larger datasets. This observation aligns with fundamental learning theory about the data requirements of deep models.

---

[1] Vincent, Pascal, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. “Extracting and Composing Robust Features with Denoising Autoencoders.” In Proceedings of the 25th International Conference on Machine Learning (ICML ’08)

We want to express our sincere gratitude for your invaluable comments and suggestions. The point-to-point responses are given as follows:

• Q1: Is it possible to design a strategy and dynamically adjust the coefficient of the three loss terms?

A1: Thanks to the comments. It is indeed possible. In our current experiments, it is shown in Figure 3 and Figure 4 of Appendix C.2 that our approach achieves relatively stable performance as the coefficients vary within a broad range. Therefore, the coefficients of the three loss terms are fixed as 1 during training. This static strategy is chosen to simplify the optimization landscape.

As for possible dynamical strategies, we may suggest adopting gradient balancing techniques [1][2] to automatically align gradient magnitudes across loss terms. This is indeed a promising direction that could further enhance our model's adaptability.

• Q2: Is it possible to evaluate the influence of dependence modeling in multi-dimensional classification tasks?

A2: Thank you for raising this critical question about evaluating the influence of dependence modeling in MDC. Your observation highlights a fundamental limitation in current methodological practices. Existing MDC approaches mainly evaluate the influence of dependence modeling implicitly through the downstream classification metrics in terms of HS, EM and SEM compared to the baseline approach (BR [3], a classic approach which ignores the dependence modeling completely).

Given the indefinability of dependence, explicit and direct evaluation methods for the influence of dependence modeling may be inaccessible. We suggest seeking possible theoretical definitions of dependence from information theory [4] and further exploring appropriate evaluation methods.

---

[1] Yu T, Kumar S, Gupta A, et al. Gradient Surgery for Multi-Task Learning[J]. 2020.DOI:10.48550/arXiv.2001.06782.

[2] Chen Z , Badrinarayanan V , Lee C Y ,et al. GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks[J]. 2017.DOI:10.48550/arXiv.1711.02257.

[3] Zhang, M.-L., Li, Y.-K., Liu, X.-Y., and Geng, X. Binary relevance for multi-label learning: An overview. Frontiers of Computer Science, 12(2):191–202, 2018

[4] Tishby N. The information bottleneck method[J]. 1999. DOI: 10.1145/345508.345578.

We want to express our sincere gratitude for your invaluable comments and suggestions. The point-to-point responses are given as follows:

• Question 1: Could you please provide more explanations for the assumptions mentioned in Eq. (3)?

Answer for Q1: Eq.(3) serves as a sufficient yet non-necessary condition for ensuring the equivalence between Eq.(1) and Eq.(2). Recall that in MDC, each feature variable $X$ is associated with $q$ class variables $Y_j (1\le j\le q)$, forming a composite class variable $Y=(Y_1,\dots,Y_q)$. Therefore, the class variable considered in Eq. (1) is $Y=(Y_1,\dots,Y_q)$, which models the joint class structure. However, existing MDC approaches mainly adopt Eq.(2) as the loss function which considers $q$ scalar class variables $Y_j (1\le j\le q)$ independently and aggregates errors across dimensions via summation (i.e., $\sum_{j=1}^{q}$ over the dimension index).

One straightforward way to impose equivalence between Eq.(1) and Eq.(2) is to assume that each summand is equivalent, i.e., Eq.(3), which actually assumes the partial conditional independence among class variables. We use “partial” here because the assumption does not require Eq.(3) to hold universally for all possible values of $Y$ and $X$, but rather only under the observed training distribution (which is finite and empirically sampled). We will make this clearer in the revised version.

• Question 2: Does Theorem 3.3 mean that the candidate set will be reduced with the mild assumptions (Assumption 3.4)?

Answer for Q2: Yes, precisely. Eq.(9) involves modeling the class space $\mathcal{Y}$, whose cardinality $|\mathcal{Y}|=\prod^{q}_{j=1}K_j$ grows exponentially with the number of dimensions $q$, resulting in a computationally intractable hypothesis space. By focusing on high-frequency class combinations, the full joint space $\mathcal{Y}$ will be reduced to a sparse subset $\mathcal{D}’$ which is computable in practical algorithms. As shown in the following table, the candidate set is reduced dramatically (with the frequency threshold $c$ set as $0.1$% when constructing $\mathcal{D}’$).

• Question 3: I am curious about whether VLMs/MLLMs can perform such tasks better than the conventional deep learning methods. An intuition here is that the powerful foundation models can directly handle the multimodal data, like the image and text labels in this paper. It would be interesting to discuss this in the paper.

Answer for Q3: We sincerely appreciate the insightful suggestion regarding the potential of VLMs and MLLMs for MDC. We fully agree that foundation models like CLIP or GPT-4V could revolutionize MDC by unifying visual and textual reasoning—a direction we are actively exploring in ongoing work. However, our current work faces a practical limitation: most existing MDC data sets (except for the exemplar data set DeepFashion) represent class labels as numerical indices (0, 1, $\dots$) rather than rich semantic descriptors. This format significantly constrains our ability to leverage the text-image alignment capabilities that make VLMs/MLLMs so powerful.

Besides, we do believe VLMs/MLLMs will likely demonstrate superior performance with more semantically-rich MDC data sets in the future and will incorporate this important discussion in our revised manuscript to better highlight both the current challenges and future opportunities in applying VLMs/MLLMs models to MDC.