Supervised Disentanglement Under Hidden Correlations

Dear Reviewer HxXe:

What an honor to engage in discussion with you again! We deeply appreciate your positive acknowledgment of our problem setting, motivation, technical quality and empirical evidence. As you have checked yourself, we take reviewers' suggestions into serious consideration and put in great effort in improving our work. We highly value your comments, and are dedicated to addressing your concerns. Accordingly, we have uploaded the latest revised version, with blue markings indicating new or modified content.

W1: Connections between theory and learning

Thank you for your detailed suggestions. We agree that a structured description would be more clear. As suggested, we have revised "Theoretical Contributions", Section 3.3 to explicitly outline the connection between theory and learning. For your convenience, the revised contents are provided as follows:

• Formally, under the independent mechanism assumption in Definition 1, given the label supervision of all attributes and modes, when the supervised losses on all attributes and modes are optimized, and the CMI of $z_k$ ($I(z_k;z_{-k}|m_k)$ for multi-modal or $I(z_k;z_{-k}|a_k)$ for uni-modal cases) is minimized, the learned $z_k$ is disentangled in the sense of Definition 2, as elaborated in Appendix B.5.

In addition, we have moved the connection between Informativeness and supervised losses to the second paragraph of "The Sufficient Condition for Disentanglement", Section 3.3 for clarity.

• Informativeness requires $I(z_1; a_1) = H(a_1)$ and $I(z_1; m_1) = H(m_1)$, which can be achieved by cross-entropy minimization [1].

Reference:

[1] A unifying mutual information view of metric learning: Cross-entropy vs. pairwise losses, ECCV, 2020.

W2: Self-contained theoretical results

We appreciate your invaluable suggestion and agree that propositions should be more self-contained. In the revised paper, we have revised the statements to ensure that each proposition and corollary is more clearly articulated, with important context and symbols restated within the propositions themselves, rather than relying on previous definitions.

For your convenience, we provide the revised version as follows:

• Proposition 1: For representations $z_1, z_2$ of $m_1, a_2$, respectively, if $I(m_1; a_2|a_1)>0$, then enforcing $I(z_1; z_2|a_1)=0$ leads to at least one of $I(z_1;m_1) < H(m_1)$ and $I(z_2;a_2)<H(a_2)$.

• Proposition 2: For representations $z_1, z_2$ of $m_1, a_2$, respectively, if $I(z_1;m_1) = H(m_1)$, $I(z_2;a_2)=H(a_2)$, and $I(z_1;z_2|m_1)=0$, then $I(z_1;a_2)=I(m_1;a_2)$ and $I(z_1;a_2|m_1)=0$.

• Proposition 3: Under the data generation assumption of Definition 1 ($K=2$, $k=1$) with independent mechanisms, if $I(z_1;a_2|m_1)=0$ for representation $z_1$, then $p(z_1| \mathrm{do} (a_{2}))=p(z_1)$.

Q1: (minor) Method Name

We gladly elaborate on the reasons for renaming as follows.

• We agree with you that SD-HC also describes our problem setting. Since we are the first to raise and solve this problem, we consider it appropriate to highlight this problem in our method name.
• Our method offers a coherent framework that involves clustering and disentangling, which can be flexibly tailored to disentangle different numbers of attributes with different forms of independence constraints according to the downstream tasks. We adopt the name SD-HC to better incorporate the overall framework and the general sense of our method.
• In contrast, the previous name M-CMI only reflects the core loss function (mode-based conditional mutual information) of our method, yet it fails to incorporate the mode label estimation part of our method and the flexibility of our method.

Thanks again for your invaluable comments. We hope our responses address your concerns and earn a more favorable perspective on our submission. Should there be further questions, we'd love to answer them!

W3.2: Theoretical link between Proposition 2 and disentanglement

In light of your comment, we provide more rigorous proof from Proposition 2 to disentanglement using do-calculous. But first, we explain why we introduce Proposition 2, and then we discuss the implications of Proposition 2 and offer a deeper understanding of disentanglement defined in Definition 2.

• Why introduce Proposition 2 from the perspective of mutual information: As agreed upon in W2 and supported in Figure 2 in [6], a causal graph with the extracted representations $z$ should contain the path $a \to x \to z$, starting from attributes $a$. Through representation learning, we cannot impose causal relationships on the extracted $z$, as $z$ is essentially learned from $x$ and inherits $a$ as its indirect causes. However, we can control what information $z$ encodes from $x$ by designing learning objectives. Thus, we adopt mutual information to evaluate the information $z_1$ encodes about the other attribute $a_2$ in Proposition 2.

• The implications of Proposition 2: We have added a conclusion in Proposition 2 and provided the proof in Appendix B.3, which is in line with your observation "if you can find the modality variable and condition on it, the associated estimated variable will be d-separated from other attributes". The added $I(z_1;a_2|m_1)=0$ implies that $z_1$ does not contain any more information about irrelevant attributes given $z_1$'s mode. This reflects the desired disentanglement property: $z_k$ should only encode $a_k$ and $m_k$, and changes in other attributes $a_{-k}$ should not affect $z_k$ if $m_k$ remains unchanged, which is formalized in Definition 2 using do-operations of external intervention. Kindly note that Definition 2 is optimized by omitting data $x$ and attribute values $\alpha_{-i}$ to better focus on the effects of intervention.

• Proving disentanglement: The definition of disentanglement in Definition 2 is a requirement on the post-intervention distribution of the extracted representations, which cannot be directly estimated due to unobserved confounders [6,7]. For rigorous proof from Proposition 2 to disentanglement, we use do-calculous to translate the post-intervention distribution with do-operators into expressions involving only observed data. The proof mainly relies on the independent mechanism assumption of data, which assumes causal independence among attributes, and the conditional independence $I(z_1;a_2|m_1)=0$, which can be enforced by minimizing mode-based CMI as shown in Proposition 2.

Our proof is given in Appendix B.4, and relevant theories are provided in Appendix C.2. We would like to express our sincere gratitude once again for your high-level academic insights, which have been greatly beneficial in improving the rigor of our theory!

Reference:

[6] Robustly disentangled causal mechanisms: Validating deep representations for interventional robustness, ICML, 2019.

[7] Desiderata for representation learning: A causal perspective, arxiv, 2021.

Your comments are greatly appreciated. During the last few days, we've been dedicated to addressing your concerns with thorough and careful consideration. We hope the clarifications and supplemented proof can address your concerns and earn a more favorable perspective on our submission. If there are any remaining issues, we'd love to address them further!

W3: From Proposition 2 to disentanglement

Thanks for this insightful comment. We'd like to address your concern from two aspects in the responses to W3.1 and W3.2.

W3.1: Discovery of modes

You are right that we cannot guarantee that the true mode labels will be discovered exactly. However, we have analyzed clustering performance and the effect of poor clustering with the following experiments:

• Various factors might impact clustering performance. We have evaluated clustering performance and analyzed the impact of noise level in "Clustering Evaluation", Appendix I. We find that although clustering performance drops as the noise level increases, k-means has been generally effective across our experiments.

• We have further analyzed the impact on attribute prediction performance in "The Impact of Noise Level and Train Hidden Correlation", Section 5.4, with the results in Figure 8ab. We find that as clustering performance drops with increasing noise levels: (1) the gap between SD-HC (based on estimated mode labels) and SD-HC-T (based on ground-truth mode labels) slightly increases then stabilizes; and (2) SD-HC remains superior in general. This indicates that although SD-HC's performance is slightly affected by poor clustering on noisy data, it still surpasses the baselines due to partial preservation of mode information.

In addition, the clustering method is a choice of design for our method, and clustering performance can be improved by choosing appropriate clustering algorithms [3,4,5] for specific applications. In light of your comment, we added discussions in "Clustering Evaluation", Appendix I for practical guidance and mentioned this in "Mode Label Estimation", Section 4. Please refer to the revised paper for details.

Reference:

[3] Marigold: Efficient k-means clustering in high dimensions, VLDB, 2023.

[4] Deepdpm: Deep clustering with an unknown number of clusters, CVPR 2022.

[5] Self-supervised convolutional subspace clustering network, CVPR 2019.

Dear Reviewer wrRV:

We sincerely thank you for the favorable recognition of the importance of the studied problem and the soundness of our experiments. We appreciate your detailed and perceptive feedback, and address your concerns with thorough and careful considerations as follows. Accordingly, we have uploaded the latest revised version, with blue markings indicating new or modified content.

W1: Coherent example

We agree that a coherent running example would greatly aid the readability. We'd like to point out that coherent examples in the context of wearable human activity recognition (WHAR) have been included throughout our paper, explaining our problem settings in the 3rd paragraph, Introduction, and the instantiation w.r.t. the specific attribute settings in the "Intuitive Example", Section 3.1.

In light of your suggestion, we add examples w.r.t. the objective of the task at the end of "Intuitive Example", Section 3.1, and further explanations on the instantiations w.r.t. the settings on real-world datasets in "Datasets", Section 5.1. For your convenience, we provide the content as follows:

• The objective of the task: For activity recognition, the goal is to learn disentangled activity representations that fully capture the activity and its modes, while remaining unaffected by personalized user patterns.

• Instantiations on real-world WHAR datasets: WHAR identifies activities with variations under each activity. Accordingly, $a_1$ represents activity, $m_1$ represents unknown activity modes, and $a_2$ represents user ID, which is often correlated with activity due to personal behavior patterns.

W2: Causal graph of representation learning

Thank you for pointing this out! We agree with you that the extracted representations should not appear in the true data-generating process. We realize that our presentation may have led to some misunderstanding, as the $z$ in Figure 3 does not indicate the representations extracted from data. Allow us to clarify this point:

• The meaning of $z$ in Figure 3: This $z$ indicates the true latent representations of attributes or modes in the underlying data generation process. The goal of disentangled representation learning is to capture the information contained in such latent representations, which is why we call them "ideally disentangled" at first (although this might be misleading). Similar data generation graphs that involve an edge from such $z$ to $x$ include Figure 1 in [1] and Figure 1 in [2].

• Intuitive example: For human activity data, we consider the edges in $a_1 \to m_1 \to z_1 \to x$, where $a_1$ represents the activity label (e.g., "walking"), $m_1$ represents the mode label (e.g., walking mode "stroll"), and $z_1$ captures the specific body movements that characterize the activity (e.g., the pace, stride, posture that identifies "stroll"). Along with the latent representations of other attributes, $z_1$ eventually generates the human activity data.

We feel the need to highlight the difference between the true latent representations and the extracted representations, although this might be implicit in relevant works [1,2]. Revisions have been made in "The Necessary Condition for Disentanglement", Section 3.3, including using distinct notations for the true latent representations ($z^l$) and the extracted representations ($z$), and explicitly explaining this in plain text.

Reference:

[1] Auto-encoding variational bayes, arXiv preprint, 2013.

[2] Hierarchical generative modeling for controllable speech synthesis, ICLR, 2019.

W1.3: Link between Proposition 2 and Definition 2

We sincerely appreciate your scientific rigor, and fully concur that Proposition 2 is a mutual information result that is not directly related to causality. As suggested, we provide more rigorous proof. In addition, we'd like to provide more insights regarding Proposition 2 and disentanglement.

• Why introduce Proposition 2 from the perspective of mutual information: As agreed upon in W1.1 and supported in Figure 2 in [4], a causal graph with the extracted representations $z$ should contain the path $a \to x \to z$, starting from attributes $a$. Through representation learning, we cannot impose causal relationships on the extracted $z$, as $z$ is essentially learned from $x$ and inherits $a$ as its indirect causes. However, we can control what information $z$ encodes from $x$ by designing learning objectives. Thus, we adopt mutual information to evaluate the information $z_1$ encodes about the other attribute $a_2$ in Proposition 2.

• A deeper understanding of the definition of disentanglement: We adopt the definition of disentanglement in [4,5], which states that if the probability distribution of $z_i$ stays unaffected by external interventions on other attributes $a_{-i}$, then $z_i$ fits the independence requirement of disentanglement. Note that this is a requirement on the post-intervention distribution of $z_i$, rather than the causal relationship. Interventions with do-operations help us determine how variables are affected by the changes of other variables, and such effects might vary for different observed data with the same causal graph structure. Kindly note that Definition 2 is optimized by omitting data $x$ and attribute values $\alpha_{-i}$ to better focus on the effects of intervention.

• Proving disentanglement: For rigorous proof, we use do-calculous to translate the post-intervention distribution of $z_i$ into expressions involving only observed data, as such post-intervention distribution with do-operations cannot be directly estimated due to unobserved confounders [4,5]. The proof mainly relies on the independent mechanism assumption of data, which assumes causal independence among attributes, and a conditional independence property of representations, which can be enforced by minimizing mode-based CMI as shown in the revised Proposition 2.

Our proof is given in Appendix B.4, and relevant theories are provided in Appendix C.2. We sincerely appreciate your profound academic insights, which have been truly inspiring in improving the rigor of our theory!

Reference:

[4] Robustly disentangled causal mechanisms: Validating deep representations for interventional robustness, ICML, 2019.

[5] Desiderata for representation learning: A causal perspective, arxiv, 2021.

Dear Reviewer dyTi:

Thank you for your careful review and constructive feedback. We deeply appreciate your positive acknowledgment of our work's novel problem setting and comprehensive experiments. We are especially grateful for your detailed comments and invaluable suggestions. Accordingly, we have uploaded the latest revised version, with blue markings indicating new or modified content.

W1.1: Theoretical explanation of Figure 3

Thank you for pointing this out! We fully agree with your professional opinion that there should be a directed edge from $x$ to the extracted representations $z$, which is a function of $x$. We realize that our presentation may have led to some misunderstanding, as the $z$ in Figure 3 does not indicate the representations extracted from data. Allow us to clarify this point:

• The meaning of $z$ in Figure 3: This $z$ indicates the true latent representations of attributes or modes in the underlying data generation process. The goal of disentangled representation learning is to capture the information contained in such latent representations, which is why we call them "ideally disentangled" at first (although this might be misleading). Similar data generation graphs that involve an edge from such $z$ to $x$ include Figure 1 in [1] and Figure 1 in [2].

• Intuitive example: For human activity data, we consider the edges in $a_1 \to m_1 \to z_1 \to x$, where $a_1$ represents the activity label (e.g., "walking"), $m_1$ represents the mode label (e.g., walking mode "stroll"), and $z_1$ captures the specific body movements that characterize the activity (e.g., the pace, stride, posture that identifies "stroll"). Along with the latent representations of other attributes, $z_1$ eventually generates the human activity data.

We feel the need to highlight the difference between the true latent representations and the extracted representations, although this might be implicit in relevant works [1,2]. Revisions have been made in "The Necessary Condition for Disentanglement", Section 3.3, including using distinct notations for the true latent representations ($z^l$) and the extracted representations ($z$), and explicitly explaining this in plain text.

Reference:

[1] Auto-encoding variational bayes, arXiv preprint, 2013.

[2] Hierarchical generative modeling for controllable speech synthesis, ICLR, 2019.

W1.2: List of assumptions

Thank you for your insightful comment. The claim "mode-based conditional mutual information (CMI) minimization is the necessary and sufficient condition for supervised disentanglement" does depend on the data generation assumptions in Definition 1, which mainly relies on the Independent mechanism assumption as follows:

• Independent mechanisms [3]: Attributes are casually independent, i.e., each attribute arises on its own, allowing changes or interventions on one attribute without affecting others. Still, confounding may exist.

However, the data generation process does not strictly depend on any assumptions in the following aspects:

• The number of modes: We use variable $m_k$ to model the possible multi-modal data distribution under a certain attribute $a_k$. This can be generalized to the uni-modal case, where the number of modes $N_m$ under each value of $a_k$ reduces to 1, and mode-based CMI degrades to attribute-based CMI.

• Correlation strengths: We use confounders to model attribute correlations and hidden correlations. Confounders may induce correlations of any strength, including the strongest correlation (indicating one-to-one correspondence) and no correlation (indicating independence). When hidden correlations are absent, our method performs similarly to A-CMI; as hidden correlations strengthen, our method becomes more advantageous, as shown in Figure 8cd and proved in Proposition 1.

We consider the above two points as advantages of our method: While mode-based CMI minimization excels under multiple modes and strong hidden correlations, the proof generalizes to uni-modal and uncorrelated cases, ensuring effectiveness across complex and simple scenarios without strict assumptions. We have made revisions to clarify this in "Data Generation Process", Section 3.1, the first paragraph in Section 3.3, and "Generalization to Multiple Attributes and Simple Cases", Section 3.3.

Reference:

[3] On Causal and Anticausal Learning, ICML, 2012.

I thank the authors for their response. However, I still have concerns regarding the theoretical aspect of the paper. In particular, while the introduction of the additional notation $z^l$ helps clarify some definitions, Definition 2 remains confusing because there is no mention of the node $z$ in the causal graph.

Furthermore, I noticed that the authors changed the definition of Disentangled Representation (Definition 2) during the rebuttal phase. This change appears to accommodate a new result presented in Proposition 3. Although the proof of Proposition 3 seems correct based on my quick review, this alteration raises concerns about the rigor of the paper. Definition 2 is a critical foundation of the paper, and the authors have cited the same reference ([Wang & Jordan, 2021]) to introduce this definition both before and during the rebuttal phase. Given these issues, I will keep my score unchanged.

We sincerely thank you for your feedback. We are glad to hear that our response clarifies the definitions of $z^l$, and deeply appreciate your effort to check the proof of Proposition 3. Your scientific rigor has inspired us.

Allow us to further address your concerns about Definition 2 regarding the causal graph and the adjustment on Equation 2.

Causal graph with the extracted $z$

In our last revised version, we include such graph in the proof of Proposition 3, Appendix B.4. As agreed in W1.1, there is an edge from data $x$ to the extracted representations $z$, which is also supported by Figure 2 in [2].

In light of your suggestion, we agree that this graph is important, and will move it to Section 3. However, we might not be able to upload the revised version in a limited time, as it requires careful adjustment.

Adjustments on Equation 2

Thank you for bringing this up. Indeed, Definition 2 is the foundation of our paper. We have shared our adjustments on Equation 2 in the response and revised paper, and welcome an open discussion.

As mentioned in our previous response, "Definition 2 is optimized by omitting data ($x$) and attribute values ($\alpha_{-i}$) (in Equation 2)". This change is part of the thorough adjustment to respond to W1.1, which involves a new definition of extracted $z$ that differentiates from the true latent $z^l$ and affects Equation 2. We elaborate as follows:

• Originally, we formulate Equation 2 with reference to [1] ([Wang & Jordan, 2021] as you mentioned), which might be misleading in not differentiating the extracted representations, the true latent representations, and the generative factors of data (i.e., attributes). This is fine in their work, as they mainly focus on the properties of the true latent.

  Specifically, Equation 5 in [1] defines representations $Z$ as disentangled if, for $j=1, ..., d$ and any value $z_{-j}$ for $Z_{-j}$,

  $P(Z_j|do(Z_{-j}=z_{-j}))=P(Z_j)$

  where their $Z$ sometimes represents the extracted representations judging by context, but also represents the factors that generate data in Definition 3 [1], and the properties of this $Z$ are derived as the necessary condition for disentanglement in Theorem 4 [1], treating $Z$ as the true latent.

  With reference to [1], although we did not explicitly differentiate true $z^l$ and extracted $z$ in the original version, we tried to imply that the $z$ in Equation 2 is the extracted representation by adding $x$ as the condition of $z$. We also explicitly differentiated $z$ from data attributes. Yet it was still unclear.

[1] Desiderata for representation learning: A causal perspective, arxiv, 2021.

• As you insightfully point out in W1.1, the true and extracted representations need to be explicitly differentiated, with which we entirely concur. During our revision, we found that this issue also affects Equation 2, which needs to be modified according to our new definitions of $z$ and $z^l$. To this end, we revised Equation 2 according to another reference [2].

  Specifically, [2] uses $G$ to represent generative factors of data (i.e., attributes), and uses $Z$ to explicitly represent the extracted representations. They define the interventional effect of a group of generative factors $G_J$ on the extracted $Z$ as:

  $p(z_I|do(G_J \gets \overset{\triangle}{g_J}))$

  where $I, J$ denotes indices of representations and generative factors, and $z_I, \overset{\triangle}{g_J}$ represent some values of $Z_I, G_J$. This formulation is based on true generative factors (attributes) and the extracted representations, which fit our revised meaning of $z$. Thereby, we adopt this definition of interventional effect and formulate Equation 2 as:

  $p(z_i|do(a_{-i}))=p(z_i)$

  where attribute values ($a_{-i}=\alpha_{-i}$) is omitted for simplicity, and the right-hand side is formulated without do-operation, meaning $z$ is unaffected by such intervention.

[2] Robustly disentangled causal mechanisms: Validating deep representations for interventional robustness, ICML, 2019.

We believe this equation clearly reflects the interventional effect of attributes on the extracted representations. Then, we developed Proposition 3 based on it. As a side benefit of your insightful comment W1.1, this revision absolutely paved the way for the link to disentanglement, which relies on a clear definition of disentanglement.

Finally, we assure you that this change does not affect any other proofs or theories. As you insightfully noticed, before Proposition 3, our theory did not directly link to Definition 2. Moreover, the meaning of disentanglement never changed, we only changed Equation 2 according to the refined meaning of $z$.

We'd like to express our gratitude again for requesting clarification. We hope this addresses your concern, and would gladly answer any further questions.

Dear Reviewer UiJJ:

Thank you for your careful review and invaluable comments. Your affirmation of our work's theoretical novelty and experimental quality inspires us to further improve our work. More importantly, we deeply appreciate your detailed feedback and constructive suggestions. Accordingly, we have uploaded the latest revised version, with blue markings indicating new or modified content.

For a better understanding, please allow us to first clarify the key concepts in W3, then address the other concerns followingly.

W3: Concepts of "data mode" and "multi-modal"

We appreciate your attention to the key concepts and we'd like to provide the following clarifications. You are right that our definitions of "multi-modal" differ from those referring to different data modalities such as text and image. Instead, we refer to the complex distributions of data. A formal definition is provided as follows, which is incorporated in "Multi-Modality and Hidden Correlations", Section 3.1 to enhance clarity.

• By multi-modal data distributions, we refer to complex data distributions that can be formulated as probabilistic mixture models [1], e.g., Gaussian mixture models $p(x)=\sum_{k=1}^{K} \pi_k N(x;\mu_k, \psi_k^2)$ with $K$ Gaussian components, where $\pi_k, \mu_k, \psi_k$ denote the weight, mean vector, and covariance matrix of each component. Samples from such distributions might originate from different components of the mixture models, concentrating around different centroids and forming different clusters.

• Data modes correspond to the components in the mixture model. Each sample originates from one component, which is indicated by its mode label. Kindly note the distinction between data modes (a property of data) and mode labels (the labels assigned to each sample based on its mode).

Reference:

[1] Minimax theory for high-dimensional Gaussian mixtures with sparse mean separation, NeurIPS, 2013.

W1: Motivation for introducing $m_k$

Regarding "hidden correlations", it seems there may be some misunderstanding. Allow us to clarify our problem setting and the different correlation types first, then explain the advantage of introducing $m_k$ with an intuitive example.

• Problem setting and difference between attribute correlations and hidden correlations: Following the common setting of supervised disentangled representation learning (DRL) [2], we assume the attribute labels are known. In this case, attribute correlations (between $a_k$ and $a_{-k}$ as you mentioned) are observable, which makes DRL under attribute correlations a more tractable problem. We assume data follow multi-modal distribution under each value of $a_k$, where data modes are unknown. Accordingly, hidden correlations are defined as the correlation between the data modes under $a_k$ and other attributes $a_{-k}$, which are unobservable with modes unknown. This is a realistic yet challenging problem that deals with underlying variations and correlations related to supervised attributes.

• Advantage of introducing $m_k$: The introduction of the mode variable $m_k$ is crucial for discovering and preserving the inherent structure of the complex data distribution under attribute $a_k$. Data modes capture the underlying variations when attribute values are held constant, and mode information can be important for attribute prediction [3]. Without explicitly considering modes, we risk losing important information for predicting attribute $a_k$.

• Intuitive example: For the activity attribute in human activity recognition, "walking" activity can involve different modes like casual "stroll" or energetic "skip walk". Without encoding these modes, a model might confuse walking modes with other similar activities, e.g., confusing "skip walk" with "climbing down" due to their similar leaping motions. Encoding modes allow the model to explicitly recognize different walking modes as "walking" activities and better differentiate them from other activities. A case study on human activity recognition has been performed in "WHAR Visualizations", Section 5.4.

In addition, we'd like to point out that our proposed method can deal with both attribute correlations between $a_k$ and $a_{-k}$ and hidden correlations between $m_k$ and $a_{-k}$, thus being inclusive rather than exclusive of attribute correlations. For theoretical and experimental details, please refer to Section 3.3 and CMNIST settings in "Evaluation Protocols", Section 5.1.

Reference:

[2] Disentanglement and generalization under correlation shifts, CoLLAs, 2022.

[3] Adaptive local linear discriminant analysis, TKDD, 2020.

W2: Mode label estimation

You are right that the performance relies on mode label estimation. We highlight the analysis we have performed on this matter, and offer more practical guidance as follows:

(1) The number of modes

• We choose the number of modes $N_m$ by hyper-parameter tuning, as stated in "Mode Label Estimation", Section 4. We analyze the parameter sensitivity in "Parameter Sensitivity of $N_m$", Appendix I, and observe that SD-HC is robust to $N_m$ when it's equal to or slightly above the ground-truth number of modes. This is mentioned in "Additional Analysis", Section 5.4.

• Alternatively, we could estimate the number of modes in a data-driven manner. This reduces the computational load of hyper-parameter tuning, but requires expert knowledge to choose the suitable method: For well-separated clusters, Elbow Method [4] would be suitable for estimating $N_m$ with k-means clustering. For complex and overlapping clusters, Bayesian Information Criterion [5] would be suitable with Gaussian Mixture Models for clustering. As suggested, we have added discussions in "Parameter Sensitivity of $N_m$", Appendix I for practical guidance and mentioned this in "Mode Label Estimation", Section 4.

(2) Challenging clustering on complex data

• We have evaluated clustering performance under varying noises in "Clustering Evaluation", Appendix I, with the results in Figure 15ab. We find that the clustering performance drops as noise level increases, because it is harder to acquire high-quality representations on complex data with large noises, as you mentioned.

• We have analyzed attribute prediction performance under varying noises in "The Impact of Noise Level and Train Hidden Correlation", Section 5.4, with the results in Figure 8ab. We find that as noise level increases: (1) the gap between SD-HC (based on estimated mode labels) and SD-HC-T (based on ground-truth mode labels) slightly increases then stabilizes; and (2) the performances of all methods drop, yet SD-HC remains superior in general. This indicates that although SD-HC's performance is slightly affected by poor clustering on noisy data, it still surpasses the baselines due to partial preservation of mode information.

• In addition, deep clustering could be explored for complex data. Deep clustering methods adopt alternative representation learning and clustering, so that the two tasks can mutually enhance each other, and sometimes incorporate self-supervised learning to improve the quality of representations [6,7]. As per your suggestion, we add discussions in "Clustering Evaluation", Appendix I for practical guidance and mentioned this in "Mode Label Estimation", Section 4.

Reference:

[4] The determination of cluster number at k-Mean using Elbow method and purity evaluation on headline news, iSemantic, 2018.

[5] A widely applicable Bayesian information criterion, JMLR, 2013.

[6] Deepdpm: Deep clustering with an unknown number of clusters, CVPR 2022.

[7] Self-supervised convolutional subspace clustering network, CVPR 2019.

W4: Typo

Thanks for your scientific rigor. As suggested, we have checked the paper and made revisions.

Q1: Multiple attributes with multi-modality

A quick answer is YES, SD-HC can address the complex scenario where any number of attributes exhibit multi-modality. We provide theoretical and implementational discussions to support this statement.

• Theoretical guarantees: As mentioned in "Theoretical Contributions", Section 3.3, our theoretical results focus on disentangling one arbitrary attribute $a_k$ among $K$ attributes, and show that one independence constraint is sufficient to disentangle $a_k$. For $a_k$ with multi-modality, this constraint minimizes conditional mutual information based on $m_k$. Similarly, for another $a_j, j \ne k$ with multi-modality, we can disentangle it the same way as $a_k$, i.e., we estimate its mode labels $m_j$ and add one independence constraint based on $m_j$ between representations $z_j$ and other representations $z_{-j}$.
• Implementational flexibility: As mentioned in the second paragraph of "Framework", Section 4, our framework can be tailored to disentangle multiple attributes with multimodality. The specific adjustments involve constructing a discriminator and mode predictor for this attribute, and adding mode prediction loss and independence constraint based on its estimated mode labels.

Thanks again for your invaluable comments. We hope our responses address your concerns and earn a more favorable perspective on our submission. Should there be further questions, we'd love to answer them!

Thank you again for reviewing our paper and raising the score! Over the past few days, we have been diligently studying your comments, gaining a deeper understanding of the issues raised. Although it appears that you require no more clarifications, we'd like to provide additional results in response to your comment Q1 (Multiple attributes with multi-modality).

Specifically, we experiment on a new toy dataset as described below, and have validated the effectiveness of SD-HC in a more complex scenario.

Data construction

The new toy data are 4-dimensional with 4 attributes ($a_i, 1 \le i \le 4$), where 2 attributes ($a_1, a_2$) exhibit multi-modality. We do not introduce multi-modality for all attributes, so that the impact on the attributes without multi-modality can still be observed.

The new toy dataset is an extension of the original toy dataset in our paper. Similar to the original toy data, each data axis is controlled by one attribute, i.e., $x_i$ is affected by $a_i$ and noises, and unaffected by other attributes $a_{-i}$. Here $a_1$ and $a_2$ with multi-modality are constructed the same as the $a_1$ in the original toy data, with 3 modes under each attribute value; $a_3$ and $a_4$ are constructed the same as the $a_2$ in the original toy data. The mappings from attribute/mode labels to the corresponding data axis are also the same.

Experiment settings

We use the same settings as the original toy data, where we train on correlated data, and evaluate on three test sets, namely test 1 with the same correlations, test 2 without correlations, and test 3 with anticorrelations. Complex attribute and hidden correlations are introduced on the train data, e.g., $I(a_2;a_4) = 0.07, I(a_3; a_4)=0.13, I(m_1;a_2|a_1)=0.14, I(m_1;a_4|a_1)=0.36, I(m_2;a_3|a_2)=0.28$, where mutual information $I(\cdot;\cdot)$ measures the correlations between variables.

The task is to learn disentangled representations for each attribute. We compare SD-HC with BASE (with supervised losses only), A-CMI, and SD-HC-T (with ground-truth mode labels). For SD-HC(-T), we use mode-based conditional mutual information (CMI) minimization for $a_1$ and $a_2$ with multi-modality, and attribute-based CMI minimization for $a_3$ and $a_4$. For A-CMI, attribute-based CMI minimization is used for all attributes.

Results

The attribute prediction accuracy is reported above, which shows that:

• A-CMI performs badly on $a_4$, even though $a_3, a_4$ do not exhibit multi-modality and are easy to predict for the BASE method. This is because under hidden correlations $I(m_1;a_4|a_1)$, attribute-based CMI minimization on $a_1$ might also hurt the predictive ability of the representations of $a_4$, as proved in Proposition 1.
• SD-HC-T outperforms SD-HC by a larger margin compared to the original toy data, probably because: (1) The increased complexity of data hinders clustering for estimating $m_i$, introducing errors via the corresponding mode-based CMI and mode prediction losses based on $m_i$; (2) The disentanglement of one representation also relies on the Informativeness of other representations, as shown in Proposition 2. Therefore, the disentanglement for $a_1$ might also be affected by mode label estimation errors on the other attribute $a_2$, which causes the loss of mode information and hurts the Informativeness of $z_2$, and vice versa.
• SD-HC generally shows superiority compared to BASE and A-CMI. For attributes $a_1,a_2$ with multi-modality, SD-HC estimates mode labels and minimizes mode-based CMI, which can preserve their mode information to some degree. For simple attributes $a_3, a_4$, SD-HC considers their hidden correlations with the modes of other attributes, which can preserve their predictive abilities.

We hope this empirical evidence further validates the effectiveness of SD-HC under more complex multi-modality, along with the theoretical and implementational aspects in our previous response. Rest assured that these additional results will be incorporated into our paper. As the discussion ends soon, we express our gratitude for your valuable comments and would greatly appreciate your continued support. Thank you very much for your time and thoughtful consideration.

We thank the area chair for the effort and support in reviewing our submission, and for sharing the new concern. We'd like to make clarifications about the new concern raised by Reviewer wrRV after the official rebuttal and discussion period, as Reviewer wrRV did not engage in discussions with us.

In short, the independence conclusions are reached based on the structural assumptions in Definition 1, and no additional parametric assumptions are needed as explained below. Also, note the distinction between causal independence and independence, which seems to have caused some confusion.

• Data generation assumption in Definition 1: Our model is built upon the data generation process of Definition 1, which includes the independent mechanism assumption that assumes attributes are causally independent. Note that this only requires that attributes are not the causal cause of one another, but allows them to be correlated due to confounding. Definition 1 includes structural assumptions about the causal graph of data generation.

• Blocking paths in Figure 3: The independence in Figure 3 is derived using causal theorems about d-separation and backdoor paths, which only rely on the given causal graph structure based on our data generation process, and no additional parametric assumptions are needed.

• Independence in Proposition 2: Proposition 2 addresses the problem that "if certain conditions are satisfied by representation learning, then what properties do the learned representations have?" The conditions "if ..." can be achieved by our learning objectives of minimizing supervised losses and conditional mutual information. Specifically, the independence $I(z_1; z_2|m_1)=0$ can be achieved using adversarial training as elaborated in "Losses", Section 4. The conclusions "then ..." are reached purely by mutual information calculations, and no additional structural and parametric assumptions are involved other than the given conditions in Proposition 2.

• In addition, we do not render $z_1^l$ independent from $a_2$. In fact, they should be correlated if attribute correlations or hidden correlations exist between $a_1, m_1$ and $a_2$. We're not sure where the confusion may arise. In line 195, we implied that they are causally independent, which means that $a_2$ is not the cause of $z_1^l$, as $z_1^l$ is the true latent representation of $z_1$.