Graph-based Unsupervised Disentangled Representation Learning via Multimodal Large Language Models

We greatly appreciate your insightful comments and commit to refining our manuscript based on your suggestions. Below, we address all your concerns.

W1: Why not replace $\beta$-VAE with advanced generative models

R1: Thanks for your insightful comments to help us improve our work. Firstly, we would like to discuss our standpoint on using other generative models, especially the Diffusion you mentioned. In our framework, the core of $\beta$-VAE based encoder $E_{sem}$ is to extract and initialize disentangled latent factors. This important perception block cannot be ideally replaced by Diffusion or other non-DRL generative models, as they do not produce an orthogonal and disentangled latent space, and thus can not comprehend and extract disentangled factors (even though some of them are also trained with variational inference).

Contrary to replacing $\beta$-VAE with the Diffusion (or other variants) as the semantic encoder, we tend to integrate it as the decoder $D_{rec}$. This integration capitalizes on the superior generative abilities of the Diffusion model to improve the reconstruction fidelity of our framework. Concomitantly, the enhanced perceptual capacity of our model can, in turn, refine the generative process of the Diffusion, thereby establishing a synergistic, mutually beneficial closed-loop. We are currently implementing this feature. However, should time constraints arise, we promise to at least include an extra discussion section to cover these content.

W2: Lacking technical novelty and the justifications of technical combinations

R2: First, we must emphasize that the technical novelty and motivation of this work is: 1) We are first to leverage the commonsense reasoning of MLLMs to discover and rank the semantic interrelations for DRL; 2) We propose a novel and practical disentanglement framework built upon β-VAE and MLLMs, with a bidirectional graph architecture, specifically designed to learn the interrelations between independent factors.

The naive combination of diffusion+MLLM could not achieve the same purpose of learning relation-aware representations, thereby facilitating practical and controllable disentanglement. This shortfall arises because Diffusion models are not designed for disentanglement, as they do not produce an orthogonal and disentangled latent space. Therefore, it is hard to directly compare our proposed method with the combinations you mentioned.

W3: Unclearness of equations and justifications in Section 3.1

R3: It's appreciated for your valuable comments. We apologize for not detailing the assumptions in the manuscript, and we promise to expand both the paper and the appendix to clarify these conditions and justifications explicitly.

Specifically, regarding the question "Is the resulting objective still convex?", we assumed that the model is upper bounded by the norm of its gradient, which satisfies the Polyak-Lojasiewicz (PL) condition, thus ensuring the suboptimality of the model. The PL condition is weaker than (strong) convexity, meaning it can be applied in a broader range of scenarios. Existing models such as L1-regularized linear regression and logistic regression have been proven to satisfy this condition, which supports our decision to use logistic regression for gradient fitting.

The assumption that $\alpha$=1 can be achieved by controlling the ratio of real samples to latent space samples. In the preliminary stage, we have tested model performance across a spectrum of hyper-parameters $\alpha$. Our empirical findings revealed a notable degradation in the quality of generated images when $\alpha$ was below 0.2 or above 5.5. Conversely, the KID metric exhibited stable consistency at $\alpha$=1 for most scenarios.

And for your last concern: Given our utilization of the discriminator for gradient fitting, the distribution of $z$ under the new objective does not affect the final optimization procedure. Consequently, we did not delve into the new distribution properties of $z$.

W4: Authors should conduct thorough ablation study to select the optimal GNN architecture

R4: Thanks for your suggestions. We will consider potentially prominent graph choices with experimental proof, provided they can be optimized within our framework in an unsupervised manner.

W5: Authors should release their code

R5: Sure, we have released our project and sent the anonymous GitHub link to the AC as required. We highly welcome fellow researchers to follow and help improve our work.

Q1: For the bidirectional weighted graph, it is unclear why this is a good choice.

R6: Thank you for the feedback. As described in Section 2.2 and Figure 1, we analyzed the reasons for proposing DisGraph for embedding knowledge instead of using a causal graph: 1) the causal relationship is often overly simplistic, typically represented as binary and impractical. However, in practice, paired variables commonly exhibit bidirectional influence, each impacting the other to varying degrees. For example, an increase in "age" can positively influence "baldness", whereas increased "baldness" does not significantly affect "age" in return; 2) Causal learning approaches are designed to model an event based on causal inference. However, our setting, akin to most DRL approaches, is to model the scenario within the observed image. When considering the interrelations between observed vision attributes, the binary causal relation are inadequate. Furthermore, we conduct the comparison results with a causal model DEAR in Section 4.1, to show the experimental superiority of our choice. Regarding your comments, we recognize that the detailed explanations of our advantages over structured DRL (Hierarchical DRL, Causal DRL, etc.) were not sufficiently clarified. We commit to refine this aspect in our revision.

We greatly appreciate all of your valuable suggestions, which play a pivotal role in enhancing the quality of our paper. Below we address all your concerns.

W1&W3: Detailed explanations of the optimization mechanism for the entire model

R1: Thanks for your feedback. We promise to enhance our manuscript with more detailed explanations of the model's optimization mechanism. Below, we offer a concise explanation of the optimization mechanism for clarity:

During the training, the optimizable parameters of the encoder $E_{sem}$, DisGraph $G$ and decoder $D_{rec}$ are denoted as $\phi$, $\gamma$ and $\theta$, respectively. And the optimization objectives can be formulated as follows:

$L_{gem}$ ($\phi$, $\gamma$, $\theta$) = $D_{\mathrm{KL}}(q_{\phi}(x, z)$,$p_{\gamma,\theta}(x, z))$, $L_{dis} = -D_{\mathrm{KL}}\left(q_{\phi}(\mathbf{z}|\mathbf{x}) \|| p_{\theta}(\mathbf{z})\right)$

$L_{total} = \lambda_{gem} L_{gem} + \lambda_{dis} L_{dis} + \lambda_{adv} L_{adv}$

where the specific optimization processes can be formulated as:

$\nabla_{\theta} L_{gem}(\phi,\gamma,\theta) \overset{x=D_{\theta}(z)}{=} -E_{z\sim q(z)}\nabla_{x}\left[\log\left(\frac{p_{\theta,\gamma}(x,z)}{ q_\phi(x, z)}\right)\right]\nabla_{\theta}x$

$\nabla_{\phi} L_{gem}(\phi,\gamma,\theta) \overset{z=E_{\phi}(x)}{=} E_{x\sim p(x)}\nabla_{z}\left[\log\left(\frac{p_{\theta,\gamma}(x,z)}{ q_\phi(x, z)}\right)\right]\nabla_{\phi}z$

$\nabla_{\gamma} L_{gem}(\phi,\gamma,\theta) \overset{z=G_{\gamma}(z_{dis})}{=} E_{x\sim p(x)}\nabla_{z}\left[\log\left(\frac{p_{\theta,\gamma}(x,z)}{ q_\phi(x, z)}\right)\right]\nabla_{\gamma}z$

Furthermore, the optimization objective of the discriminator can be expressed as:

$L_{adv} = L(D) = \frac{1}{N} \left[ \sum\limits_{i=0; (x_i, z_i) \in E_\phi}^{N} \text{softplus}(-D(x_i, z_i)) + \sum\limits_{i=0; (x_i, z_i) \in D_\theta}^{N} \text{softplus}(D(x_i, z_i)) \right]$

where $D^*(x,z)=\log\left(\frac{p_{\gamma, \theta}(x,z)}{q_{\phi}(x,z)}\right)$. The discriminator $D$ can be used to fit $D^*$ to achieve gradient estimation and complete the training. You may also check the R1and R3 for Reviewer d3xR, for insights into the framework's definitions and workflow.

W2: A brief explanation of some methods used, such as Somers' D algorithm, will be beneficial

R2: Thanks for your suggestions. We will meticulously integrate comprehensive definitions and clarifications of all employed functions. Specifically, for the Somers' D algorithm you referenced, we provide the subsequent example to enhance your understanding. Suppose we have the sample dataset $S=\{(1,2),(3,1),(2,3)\}$:

Somers' D indicator can be calculated as follows:

$D =\frac{N_c-N_d}{N_c+N_d+T_y} = \frac{1 - 2}{1 + 2 + 0} = -\frac{1}{3}$

This obtains a value of approximately -0.33, signifying a negative correlation between variables $X$ and $Y$. This calculation demonstrates that the Somers' D metric is straightforward to calculate and is particularly applicable to ordinal variables. Furthermore, Somers' D is asymmetric and capable of distinguishing bidirectional relationships between variables. These characteristics make it highly suitable for integration into our model.

Q1: How is the effectiveness of DisGraph ensured? What problems could arise if errors are introduced into DisGraph?

R3: To ensure the effectiveness of DisGraph, we perform an ablation experiment by disabling the DisGraph (see Figure 7 in Section 4.5). Since completely removing DisGraph is infeasible due to its role in embedding bidirectional weighted relations, we alternatively utilize an initial version of DisGraph without the updating module $E_{gnn}$. Refer to the top-left in Figure 7, where the model, using an initial Graph, exhibits weakened or inaccurate relational awareness (e.g., the relation between "Bald" and "Gender" weakens). This observation demonstrates the effectiveness of DisGraph.

Regarding your second question, to our knowledge, the errors in interrelation identification from MLLMs may lead to counterfactual outcomes. For example, applied with a less capable MLLM that does not recognize the positive correlations between "banana" and "yellow" or "age" and "white hair", the model may obtain a red banana or a white hair baby learned from the errors introduced into DisGraph (refer to the counterfactual results by a weaker MLLM in Figure A3 of the attachment). In our perspective, even advanced MLLMs (e.g., GPT-4o) can exhibit a certain bias. To enhance robustness of GEM, we are investigating potential improvements both within the MLLM (e.g., data pre-processing, bias-aware modules, and knowledge editing) and externally (e.g., weight redistribution, debiasing modelling, and modifications to decoding strategies).

We value your insightful feedback and will refine our manuscript accordingly. Here, we address each of your concerns.

W1: Detailed definitions of the model

R1: Thanks for your suggestions. We will detail the model's parameters, definitions, and optimization strategies in revision. Here, we present a brief overview of for clarity:

where $pre$ is the pre-defined dimensionality for all latent variables. During training, the optimizable parameters of the encoder $E_{sem}$, DisGraph $G$ and decoder $D_{rec}$ are denoted as $\phi$, $\gamma$ and $\theta$, respectively. And the optimization objectives can be formulated as follows:

$L_{gem}$ ($\phi$, $\gamma$, $\theta$) = $D_{\mathrm{KL}}(q_{\phi}(x, z)$,$p_{\gamma,\theta}(x, z))$, $L_{dis} = -D_{\mathrm{KL}}\left(q_{\phi}(\mathbf{z}|\mathbf{x}) \|| p_{\theta}(\mathbf{z})\right)$

$L_{adv} = L(D) = \frac{1}{N} \left[ \sum\limits_{i=0; (x_i, z_i) \in E_\phi}^{N} \text{softplus}(-D(x_i, z_i)) + \sum\limits_{i=0; (x_i, z_i) \in D_\theta}^{N} \text{softplus}(D(x_i, z_i)) \right]$

$L_{total} = \lambda_{gem} L_{gem} + \lambda_{dis} L_{dis} + \lambda_{adv} L_{adv}$

In the revised total loss, as distinct from the manuscript's version, we partition the original $L_{dis}$ into two components: $L_{gem}$ for reconstruction and $L_{dis}$ for disentanglement.

W2: Missing details in Figure 2

R2: Thanks for your thorough review. We have updated Figure 2 to highlight the input distribution of $D_{rec}$ accordingly.

W3_A: How to make it differentiable for end-to-end training in the model

R3: Based on the latent variable $z_{dis}$ from encoder $E_{sem}$, we generate $n$ nodes, each representing one of the $n$ semantic attributes. In practice, each node mirrors $z_{dis}$'s dimensionality but isolates the $i$-th attribute by masking all dimensions except the $i$-th, thereby to learn the $i$-th attribute. These nodes, combined with interrelations from $P_{rel}$, form the DisGraph, which outputs the embedding matrix $T$. This matrix is utilized to calculate $z$ by averaging each volume, subsequently decoded by $D_{rec}$ to reconstruct the image with our loss functions. All aforementioned modules are differentiable.

W3_B: Unclearness of the Graph learner training

R4: As described in Line 195, we follow the instructions by Liu et al. to unsupervisedly train the graph learner. Specifically, the adjacency matrix are updated within $E_{gnn}$, by the Structure Bootstrapping Mechanism and Multi-view Graph Contrastive Learning.

W3_C: Unclearness in Figure 2 about Landmark

R5: In this work, we only apply off-the-shelf landmark models for data pre-processing, which does not require any extra data in the databases. We will accordingly clarify it in Figure 2.

W4: Disentanglement metrics

R6: Our research emphasizes incorporating attribute interrelations, where enhancing disentanglement is not our primary objective (refer to R4 to Reviewer qyA4). However, to address your concern, we have conducted extra experiments as shown in Figure A4 in the attachment.

W5: Explanations of the loss function

R7: We have detailed our loss functions in R1. Regarding your final concern, our approach addresses DRL’s limitations by: 1) contributing to a more practical paradigm; 2) balancing reconstruction and disentanglement abilities by adjusting $\lambda_{gem}$ and $\lambda_{dis}$.

Q1: Is Equation (2) the fundamental objective of vanilla VAE?

R8: Equation (2) represents the objective function of the $\beta$-VAE, while Equation (1) represents the likelihood estimation for vanilla VAE.

Q2: Justifications of parameters $\theta$

R9: We adopt the original VAE notations: $p_{\theta}(z)$ implies $z$ follows a standard normal distribution without parameters. In $p_{\theta}(z|x)$ and $p_{\theta}(x|z)$, $\theta$ denotes decoder parameters. $p_{\theta}(z|x)$ is used solely for theoretical derivation of the variational lower bound, with no parameter sharing in practice.

Q3: Why can MLLMs provide accurate scores?

R10: Theoretically, the impressive capabilities of MLLMs, especially in realistic AI generation, reveal that they can comprehend real-world concepts at a certain level, informed by in-depth studies (see Section 2.3); Experimentally, our evaluations of SOTA MLLMs confirm their reliability (see Section 4.4 and Figure A1 in attachment).

Q4: Reliance on unsupervised DRL

R11: Refer to the R2 for Reviewer qyA4, the DRL branch extracts and initializes latent factors. Although these factors exhibit partial independence, this limitation is acceptable as their interrelations will be refined and updated by the MLLM branch and DisGraph. Hence, our model does not require any external supervision or priors.

Q5: Explanations on interrelations determining

R12: For determining the interrelationships between two attribute scores from MLLMs, we employ correlation analysis algorithms (e.g., Somers' D). Due to spatial constraints, please refer to R2 for Reviewer SQvd for the details of this part.

Q6: Potential applications

R13: Our practical and relation-aware DRL framework can be applied in domains like AI-generated content, explainable AI, medical imaging, robotics, and autonomous vehicles.

We greatly appreciate your insightful comments and commit to refining our manuscript based on your suggestions. Below, we address all your concerns.

W1: Dependence on MLLM

R1: Thanks for the insightful comments. First, we would like to clarify that our model leverages the commonsense knowledge embedded in MLLMs to predict interrelations. This is predicated on the assumption that MLLMs, including their future iterations, are powerful and reliable enough to comprehend the physical rules of the real world (e.g., aging brings wrinkles, sunrise brings light, etc.). To substantiate the reliability, we evaluate the scores from SOTA MLLMs with ground truths, as detailed in Section 4.4 and Figure A1 in the attachment.

Nevertheless, we totally agree with your comments, that a solitary score is not convincing enough to represent MLLMs' knowledge. Towards it, two perspectives of considerations are undertaken: 1) enhancing the prompting guidelines to have more informative outputs (e.g., likelihood probabilities, multi-level categorical outputs, averaged iterative scores and etc.); 2) interpreting lightweight bias mitigation modules with certain prior knowledge (e.g., [1] and [2]).

[1]Hauzenberger et al., Modular and on-demand bias mitigation with attribute-removal subnetworks, 2023.

[2]Kumar et al., Parameter-efficient modularised bias mitigation via AdapterFusion, 2023.

W2: Lack of Detailed Attribution

R2: Thanks. Actually, it is a frequent question why unsupervised DRL models possess the ability to align specific attributes. We provide a brief discussion here and will offer a comprehensive explanation in the revision.

To address your concern, we need to analyze it from the information-theoretic perspective. As per the Information Bottleneck (IB) theory [3], constraining the information input to the DRL model (e.g., by the penalty coefficient $\beta$ in $\beta$-VAE), inherently enables the model to identify and learn the most representative factors for successful reconstruction. For instance, when trained on the Shapes3D (a collection of simple, synthetic objects) with a merely three-dimensional latent variable, the model tends to learn the most critical factors which are observed to be "object colour", "object shape", and "background shape". These attributes are aligned in the three dimensions, organized in order of their contribution to reconstruction. Similarly for facial images, the model spontaneously learns and organizes the most informative attributes (e.g., hair, gender and etc.) in the disentangled dimensions.

In addition, in our framework, we employ a landmark detection for pre-processing, extracting pivotal object features and discarding image redundancies via cropping and resizing. It further helps the mapping of each latent representation to unique attributes.

[3] Burgess et al., Understanding disentangling in $\beta$-VAE, 2018.

Q1: I did not fully understand how the paper finds the correspondence between the latent representations output by the Encoder and the semantic attributes

R3: In the response for W2, we have clarified that for a collection of data, the most informative attributes will be unsupervisedly learned by the $\beta$-VAE and sequentially aligned across several dimensions. It highlights the role of $\beta$-VAE branch to disentangle and initialize the attributes. Nevertheless, these initialized attributes need a further human observation to extract textual concepts for subsequent MLLM prompting.

Q2: Is there a specific principle guiding the MLLM to score the attributes? For instance, what is the difference when the MLLM scores the same attribute as 2, 3, or 4? What would the results be if other MLLMs were used for scoring?

R4: In the paper, Figure 3 illustrates the specific guiding principle, which prompts MLLMs to classify one attribute into multiple degrees on its own judgement. Specifically, it assigns scores ranging from 0 to 5 for each attribute, where 0 indicates the attribute’s absence, and 5 denotes its highest expression. For example, for scoring attribute ''smile", MLLMs tend to assign a score of 0 for the absence of a smile and a score of 5 for a full laugh, according to the statistical distribution they have learned from extensive data.

Towards your 2nd question, we wish to further clarify: since the goal of the MLLM branch in GEM is to discover the interrelations, the statistical relativity between two attributes is of primary concern, rather than the absolute scores for the individual attribute (see Figure A2 in the attachment). For example, given a collection of facial images, it is acceptable if the scores of "age" and "bald" exhibit a positive correlation, even if the specific score values are fluctuating.

Addressing your final concern, we believe that a less capable MLLM would hinder the learning of attribute interrelations. We substantiated this by replacing GPT-4o with the inferior GLM and assessing the outcome. Figure A3 in the attachment demonstrates that GLMs' limited perceptual capacity yielded counterfactual outcomes. Therefore, the MLLM bias mitigation methods mentioned in R1 are valuable.

We greatly appreciate your insightful comments and commit to refining our manuscript based on your suggestions. Below, we address all your concerns.

W1: The paper would benefit significantly from clearer writing, organization and explanations

R1: Thanks for your suggestion, we promise to enhance the manuscript on the writing style and structural organization. In addition, we will ensure that all technical concepts presented in the paper are associated with clear and comprehensive definitions or derivations (e.g., Somers’ D algorithm).

W2: The heavy reliance on the disentanglement abilities of β-VAE, which itself often presupposes some level of independence or weak dependence.

R2: Thanks for your insightful comments. We wish to re-emphasize the key of our work: To move beyond the unrealistic independence assumption of traditional DRL, we propose a more logical and practical DRL paradigm that involves the bidirectional interrelations between attributes. In this framework, $\beta$-VAE branch is only employed to extract and initialize latent factors. The weak dependence or partial independence among initial factors is acceptable, as their interrelations will be subsequently determined, overwritten, and updated by the proposed MLLM branch and DisGraph. In this way, our method aligns more closely with real-world dynamics and offers broader applicability.

W3: Reliance on pre-trained multimodal large language models might introduce biases

R3: Our framework is based on the belief that MLLMs, including their future iterations, are sufficiently robust to comprehend logical rules of reality (e.g., aging brings wrinkles, sunrise brings light, etc.). These rules are represented as interrelations between entities. On this basis, we have corroborated the efficacy of MLLMs using ground truths to affirm their effectiveness in straightforward scenarios (see Section 4.4 and Figure A1 in the attachment). However, we totally agree with you that even the most powerful MLLMs (e.g., GPT-4o) can exhibit bias on limited pre-training data. To address this, we are exploring potential solutions from both within the MLLM (e.g., data pre-processing, bias-aware module, knowledge editing, and etc.) and from our end (e.g., weight redistribution, debiasing modelling, decoding strategy modification and etc.).

W4: A deeper theoretical analysis of why and how the inclusion of interrelations leads to better disentanglement would be valuable

R4: Thanks. It is really meaningful and interesting to discuss "what is a better disentanglement". If it means the better performance on independently decomposing factors, then the inclusion of interrelations might not seem beneficial; however, if it refers to a better performance/practicality for real and complex scenarios, our disentanglement paradigm excels by statistically capturing the logical rules of real world. Specifically, the inclusion of interrelations can be beneficial in model generalizability, counterfactual reasoning and practical usages. We will further conduct analysis and discussion on this point in the manuscript, through both theoretical analysis and experimental investigations.

Q1: How are the generated node embeddings T generated by DisGraph associated with the losses?

R5: Our apologies for the unclear descriptions. In brief, DisGraph generates the embedding matrix $T$ based on updated parameters to derive the latent variable $z$ through the aggregation of node embeddings. $z$ is subsequently decoded by $D_{rec}$ to reconstruct the image within the bounds of our loss functions. We welcome your reference to the R1 for Reviewer d3xR for a comprehensive discussion of the model's loss functions and workflow.

Thank you for your continued efforts. We have provided comprehensive rebuttals and tried to address the concerns raised in your reviews. Please take the time to review if possible, if you have any further questions or require additional clarification, please let us know and we welcome discussions in any format . Thanks again.