Unifying Disentangled Representation Learning with Compositional Bias

Q11. Clarification on Compositional Consistency Loss

A11. We acknowledge that our explanation may have been misleading. The reviewer’s understanding is correct. The issue we describe is not posterior collapse in the traditional sense. Rather, it refers to a scenario where reconstruction is successful ($(L_{\text{diff}}$ is very low), but $\mathbf{z}^i$ and $\mathbf{z}_j$ become close in the latent space for every data index $i, j$. In this situation, even if $\mathbf{x}^c$ generates an image irrelevant to $\mathbf{z}^c$, the penalty from $d(\mathbf{\hat z}^c, \mathbf{z}^c)$ remains low, reducing the effectiveness of the compositional consistency loss. To effectively penalize such cases, we introduce a contrastive term to ensure that $\mathbf{\hat z}^c$ remains close to $\mathbf{z}^c$ but as different as possible with respect to other negative samples $\mathbf{z}_j$. We have updated the main paper to clarify this point and better explain how our method addresses this issue.

Q12. Motivation of gradient truncation trick

A12. As diffusion models require iterative denoising steps to decode $\mathbf{z}^c$ into $\mathbf{x}^c$, it is computationally prohibitive to back-propagate the gradients through all of the denoising steps. To address this, we draw inspiration from recent works [6, 7] in diffusion-based optimization. These studies demonstrate that truncating gradients at the last iteration of the denoising process effectively balances computational feasibility and optimization performance. Following this approach, we truncate the gradient at the last iteration of the denoising step to ensure efficient back-propagation of the gradient. We added this detail in the main paper.

[1] Rombach et al., High-Resolution Image Synthesis with Latent Diffusion Models, in CVPR 22.

[2] Watters et al., Spatial broadcast decoder: A simple architecture for learning disentangled representations in vaes, in Arxiv 19.

[3] https://huggingface.co/CompVis/ldm-celebahq-256

[4] Poole et al., Dreamfusion: Text-to-3d using 2d diffusion, in ICLR 2023.

[5] Ho et al., Denoising Diffusion Probabilistic Models, in NeurIPS 2020.

[6] Clark et al., Directly fine-tuning diffusion models on differentiable rewards, ICLR24.

[7] Aligning Text-to-Image Diffusion Models with Reward Backpropagation, ArXiv.

Q7. What is the difference from the prior term in Jung et al. and how does the proposed framework compare quantitatively to Jung et al.'s approach?

A7. We appreciate the thoughtful comments. To address the first question, our frozen diffusion model is an “unconditional” model that estimates $p(\mathbf{x}^c)$ and does not rely on $\mathbf{z}^c$ during estimating the likelihood. In contrast, L2C (Jung et al.) uses a conditional diffusion model that estimates $p(\mathbf{x}^c|\mathbf{z}^c)$, and thereby likelihood estimation is directly conditioned on $\mathbf{z}^c$. Such distinction is crucial because estimation of $\log p_\psi(\mathbf{x}^c |\mathbf{z}^c)$ is inherently sensitive to the conditioning variable $\mathbf{z}^c$. In L2C, the conditional likelihood estimator $p_\psi(\mathbf{x}^c | \mathbf{z}^c)$ is learned using denoising losses with $\mathbf{z}$ encoded from individual images $\mathbf{x}$. When $\mathbf{z}^1$ and $\mathbf{z}^2$ are randomly composed to form $z^c$, the resulting $\mathbf{z}^c$ may become out-of-distribution (OOD) samples (i.e., unseen sample during training). Consequently, the estimation of $\log p(\mathbf{x}^c | \mathbf{z}^c)$ becomes inaccurate for OOD $\mathbf{z}^c$, and also there is no guarantee that maximizing $p(\mathbf{x}^c | \mathbf{z}^c)$ for OOD $\mathbf{z}^c$ yields realistic samples for $\mathbf{x}^c$. In contrast, our method employs an unconditional diffusion model pre-trained to estimate $p(\mathbf{x}^c)$ and it ensures robust estimation of $\log p(\mathbf{x}^c)$ regardless of $p(\mathbf{z}^c)$. Therefore, our prior term is more robust for maximizing the likelihood of $\mathbf{x}^c$ compared to that of L2C.

Nevertheless, we agree that comparing our method to L2C ’s approach would strengthen the analysis. To address this, we conducted experiments on CLEVRTex replacing our prior loss with the one proposed in L2C. The results are presented in Table below. Those results clearly highlight the superior performance of our prior term over L2C. We hope this explanation and comparison clarify the advantages of our approach.

Q8. Elaboration on why PCA is used when computing DCI and the resulting values are fair

A8. We did not apply PCA to the entire latent representation. Instead, PCA was performed on each vector $\mathbf{z}_i$, and the principal component with the highest principal value was selected. This adaptation was necessary because DCI is typically computed dimension-wise, but we cannot treat vector representations as scalars. By extracting the dominant component for each $i\mathbf{z}_i$, we can compute DCI while maintaining a fair evaluation across methods. We hope this explanation resolves the concern.

Q9. Standard Deviation of experiments.

A9. We appreciate the valuable comment. In attribute disentanglement, we measure the performance for 10 different runs following DisDiff. Also, following the reviewer’s suggestion, we train our model with three different seeds and report the standard deviations in Table 19 of Appendix A.9. Due to the limited time for rebuttal period, this was conducted only for our method, but we will include standard deviations for all baselines as well in the final version of the paper.

Q10. Modify Figure 1 to prevent confusion with mixing examples

A10. We appreciate the valuable suggestion. Following the suggestion, we modified the figure to clearly indicate that the illustrative example represents the object mixing strategy by adding a labeled arrow and explicitly clarifying it in the caption. While we first considered including examples for both attribute mixing and object mixing as suggested, the limited space in the figure made it challenging to accommodate both.

Q4. Could the proposed method's choice of mixing strategy be considered an inductive bias, similar to prior methods, and could state-of-the-art approaches also be adapted to disentangle both attributes and objects with modifications?

A4. We appreciate the valuable comments. We clarify that our goal is not to eliminate inductive biases specific to factors of variation but to propose an inductive bias that is compatible with disentangling multiple factors of variation (e.g., attributes and objects). Our method implements this bias in the form of a mixing strategy, which can flexibly disentangle either objects or attributes by adjusting only the mixing strategy, maintaining the same model parameterization and objective functions. In contrast, prior methods typically embed their inductive biases into the objective functions (e.g., information-theoretic objectives) or architectural designs (e.g., slot-attention encoder). It is non-trivial to adjust such objective functions or architectural designs to achieve both attributes and objects within a single framework and to the best of our knowledge, there are no successful methods for such adaptation.

For instance, information-theoretic objectives proposed in attribute disentanglement often aims to reduce statistical dependencies among latent variables. However, extending this objective function to disentangle objects in object-centric scenes—where the number of objects varies or identical objects appear in different spatial locations—is non-trivial and no such work has been reported. On the other hand, object-centric learning relies heavily on architectural biases that enforce spatial exclusiveness, which cannot naturally handle spatially non-exclusive attributes (e.g., color, shape, or texture). It is also not trivial how we should design architectural biases to promote disentanglement of spatially non-exclusive attributes.

In the same context, state-of-the-art methods like Jung et al. (2024) cannot simply disentangle both attributes and objects by simply modifying one component, such as the objective function, architecture, or mixing strategy; Specifically, (1) Even if the objective function or mixing strategy is adjusted, the slot-attention encoder inherently enforces spatial exclusiveness of factor, preventing disentanglement of spatially non-exclusive attributes. (2) Even when removing the slot-attention encoder, reuse of diffusion decoder as a generative prior hinders accurate likelihood estimation (see response to Q7) and absence of a compositional consistency loss further hinders effective attribute disentanglement (see Ablation Study in Table 3). We hope this explanation clarifies the clear contribution of our method and why prior methods have challenges in achieving both attribute and object disentanglements.

Q5. It would be great to show qualitatively in examples like Figure 2 what happens when the image contains 2 identical objects and one of them is added or removed from the image.

A5. We appreciate the valuable comments. We would like to clarify that in Figure 3, the observed issue was not a confusion by our model but rather a misdrawn arrow for the target object to be inserted. We corrected the figure in the current version. Regarding the scenario of inserting or removing objects in a scene with multiple identical objects, we agree this is an intriguing question to explore. As our learning objective encourages the model to learn "compositional" concepts, we expect it to assign each object to distinct slots, even in scenes with multiple identical objects. We will conduct this experiment and include the results in the final version of the paper.

Q6. Details of w(t) in Equation 5.

A6. We appreciate the valuable feedback. $w(t)$ is a timestep-dependent function derived by [4] and usually set to $\sigma^2_t=1-\bar\alpha_t$, where $\bar\alpha_t$ is a hyper-parameter controlling the noise schedules in the diffusion model [5]. We add this information in the main paper.

Q3. Could you provide results on at least a couple of the more complex datasets?

A3. We appreciate the valuable comments. Following the reviewer’s suggestion, we conducted additional experiments on CelebA-HQ for attribute disentanglement and MultiShapeNet for object disentanglement, respectively. Due to limited time in the rebuttal period, we conduct experiments on CelebA-HQ instead of CelebA, as we can employ a pretrained, off-the-shelf diffusion model ([3]) for the CelebA-HQ dataset. Experimental details and results are included in Appendix A.8.

For the CelebA-HQ dataset, we use the attribute-mixing strategy to disentangle attribute factors. To verify the disentanglement of the learned representations, we swap each latent vector one by one between two images and present the resulting composition images in Figure 6 and 7. In the third column of each figure, we observe that while the source images lack bangs, the swapped images successfully generate bangs while preserving other attributes. Similarly, in the fourth and fifth columns, the facial expressions (e.g., smile) and skin tones of the target images are effectively transferred to the source images. These qualitative results demonstrate that our attribute-mixing strategy is capable of disentangling attribute factors, even in complex datasets like CelebA-HQ.

We also validate our method on MSN dataset with object-wise manipulation and unsupervised segmentation. For the object-wise manipulation task, we encode pairs of images into $N=5$ object representations and exchange random object latents between the pairs to construct composite images. As shown in Figure 8, our method successfully performed object-level insertion and removal, demonstrating that each latent representation distinctly captures individual objects. This confirms that our approach effectively disentangles object representations within the latent space.

For the unsupervised segmentation task, we measure FG-ARI, mIoU, mBO on object masks following common practices in object-centric literature. As our method does not have a built-in mechanism to directly express group memberships between pixels, we additionally train Spatial Broadcast Decoder on the frozen latent representations to predict explicit object masks for each latent representation (please refer to A2 and appendix A.8 for details). The results are reported in Table 17 in Appendix A.8. Among the competitive slot-attention based baselines, our method achieves second-best performances across all of three metrics. The high segmentation scores of L2C are mainly due to its slot-attention-based regularization term (see Equation 8 in the L2C paper), which explicitly encourages the slot masks to align with object shapes. Excluding L2C, our method outperforms rests of the baselines (LSD, SLATE) across all metrics, despite not employing a spatial clustering mechanism like slot attention. These results demonstrate the effectiveness of our framework in disentangling object representations in a complex dataset.

We thank the reviewer for valuable comments and suggestions. We have revised our paper and provide clarification following the reviewers’ writing suggestions. Below we respond to the individual questions.

Q1. Could you explain or correct the mismatch between your results and those previously reported?

A1. We appreciate the valuable comments. The difference in reported metrics between baseline results is due to different experimental setups. In LSD's original experiments, the encoder $E_\theta$ directly encodes RGB images into slot representations, whereas in our experiments, encoder $E_\theta$ gets latent features obtained from a pretrained vae [1] as input. This difference comes from our effort to align input and output formats for all baselines to ensure fair and direct comparisons across methods. However, prior works on object-centric learning adopt diverse input and output formats (RGB image, VAE feature, DINO feature etc) across different models, which hinders direct comparison across the methods. To address this, we aligned both the input and output formats to latent features encoded by a pretrained vae [1] for all methods. Despite this modification to latent encoder for baselines, the results in Table 15,Table 16 and Figure 5 in Appendix A.7 consistently demonstrate that baselines still effectively learn object-centric representations under this standardized setup.

To further address the concerns, we additionally trained our method using an RGB image encoder (the only modification) identical to LSD and L2C, and reported the property prediction results in the Table below. For comparison with both LSD and L2C, we followed the evaluation protocol of the L2C paper and compared our results to the values reported in the L2C paper. As shown in Table below, our method still achieves comparable results to state-of-the-art baselines, re-ensuring that our novel mixing strategy provides a robust inductive bias for learning object-centric representations. We hope our explanation along with the additional experiments showing comparable performance to baselines, addresses the reviewer’s concerns.

Q2. Could you provide results on unsupervised segmentation tasks using FG-ARI, mIoU, mBO

A2. We appreciate the insightful comments. Following the reviewer’s suggestion, we additionally evaluated unsupervised segmentation quality of pretrained encoder and reported it in Appendix A.7. Unlike slot-attention-based methods, our method does not have a built-in mechanism to directly express group memberships between pixels. Therefore, we trained a Spatial Broadcast Decoder [2] on top of frozen latent representations to predict explicit object masks for each latent representation. We train Spatial Broadcast Decoder with a reconstruction loss to recover the original image from frozen latents in an unsupervised manner, and it requires minimal training costs as the encoder remains frozen and the decoder is shallow. With this explicit object mask, we compare our method against two strong baselines in slot-attention-based works, LSD and L2C, on CLEVR and CLEVRTex. For a fair comparison, we evaluate the baselines using both slot-attention mask and object masks obtained by training a Spatial Broadcast Decoder on their frozen slot representations. The results are reported in Table 15,16 and Figure 5 in Appendix A.7.

On the CLEVR , our method achieved the best mIoU, mBO scores and comparable FG-ARI. A high FG-ARI of our method indicates that each mask captures complete objects, confirming effective object disentanglement of our method. However, we observed that the background mask is split across multiple latents. This is because constant backgrounds in CLEVR does not affect compositional generation and thereby avoiding penalties from the compositional loss. Since the constant background carries minimal information, it does not impact the quality of object representations and compositionality, and we do not consider it a problem from an object-centric representation perspective. In the CLEVRTex, our methods outperforms both LSD and L2C for all three metrics. In Figure 5, we observed that our method consistently encodes complete objects into distinct latents, whereas LSD and L2C often split objects into multiple latents. Also, in contrast to CLEVR, as CLEVRTex has various background colors, our model successfully encodes all of the background information into a single latent. These experiments on unsupervised segmentation confirm that our pretrained encoder achieves effective object-wise disentanglement. Notably, our method outperforms baselines in object segmentation without relying on spatial clustering architectures like slot attention.

We thank the reviewer for valuable comments and suggestions. We have revised the paper to correct the typos and provide clarifications. Below, we respond to each of the individual questions.

Q1. Proposed method ensures composite image realism, but this doesn’t guarantee alignment with the intended attribute or object combinations.

A1. We appreciate the valuable comment. As the reviewer pointed out, optimizing solely for realism of composite images may produce realistic images that do not align with the source attribute or object representations. We addressed such misalignment with compositional consistency loss. This loss explicitly enforces the accurate reconstruction of source latent representations from the generated composite images. By penalizing composite images that are realistic but fail to match the source latent representations, this loss ensures alignment between realistic composite images and intended latent compositions. We hope this clarifies our approach and addresses the reviewer’s concern.

Q2. There are no guarantees that the latent representations are identifiable under the current model, and by implication, neither are the compositions.

A2. We agree with the reviewer that our method does not provide theoretical guarantees for the identifiability of latent representations or their compositions. While theoretical guarantees would certainly strengthen our work, the primary focus of our paper is to demonstrate that the incompatible inductive biases traditionally used for disentangling attributes and objects can be replaced with a unified and compatible inductive bias in the form of a mixing strategy. Furthermore, guaranteeing the identifiability of latent representations typically requires strong assumptions on the decoder function (e.g., [1,2]) or latent priors (e.g., [3,4,5]). As a result, much of the prior work in disentangled representation learning and object-centric learning often focus more on empirical performance rather than guaranteeing identifiability. Since our work prioritizes serving as a proof of concept, we also have adopted an empirical approach. Nevertheless, we agree that investigating when and how our framework can effectively learn object-centric representations and disentangle attributes within the context of identifiability theory would be a valuable direction for future research.

Q3. The fixed mixing strategies, although appropriate for the simple cases studied, are quite rigid and likely would not adapt well to more complex scenarios in real data.

A3. We appreciate the valuable comment. We believe that the “fixed” mixing strategy could adapt to more complex scenarios as well. The role of mixing strategy is to define a specific form of compositional structure that the latent representation should satisfy. By controlling the mixing strategy, the model would discover different factors of variation aligned with the intended compositional structures. For instance, when an attribute mixing strategy is applied in real-world scenes, the model would learn to disentangle unique and composable factors that are always present in the scene (e.g., global lighting or style). Conversely, applying object mixing strategies to same scenes would guide the model to disentangle dynamically occurring object components within a scene. Therefore, we believe that the fixed mixing strategies are not a significant limitation and are applicable to both simple and complex scenarios.

Q6. Could dynamic/learned mixing strategies replace fixed ones to improve flexibility in complex scenes?

A6. We appreciate the insightful question. Yes, we believe dynamic or learned mixing strategies would be an interesting and effective extension of our work to flexibly discover various factors of variations with complex compositional structures. When we apply a fixed mixing strategy, the model will consistently discover factors of variation satisfying the specific compositional structure we embed through the mixing strategy. However, in real-world scenarios, the compositional structure of the factors may vary across scenes. In such cases, applying a mixing strategy adaptive to the given scene would help capturing scene-dependent factors of variations. Moreover, learning valid mixing strategies directly from data to uncover the underlying compositional structures would be another promising future direction.

Q7. Have the authors thought about under which conditions their method can provide identifiability guarantees?

A7. Guaranteeing identifiability generally requires imposing strong restrictions on the class of decoders or the latent distribution. In recent object-centric literature, additive decoders have been widely adopted to guarantee identifiability [2, 6, 7]. For object disentanglement, our method could leverage an additive decoder as well and define compositionality using compositional contrast as proposed in [6, 7]. These steps would be the first step to provide identifiability guarantees for object representations. However, for factors of variation like attributes, which globally affect the image, the additive decoder and compositionality definitions may not be satisfied in general. In such cases, we believe that ensuring identifiability would require imposing specific structural constraints on the latent distribution, as explored in [3, 8]. Therefore, it is challenging to immediately identify a unified set of conditions that guarantee identifiability for both attributes and objects simultaneously. A promising direction would involve first establishing conditions for identifiability for attribute and object separately, and then develop a general theory to integrate these conditions.

[1] Brady et al., Provably Learning Object-Centric Representations, in ICML 23.

[2] Lachapelle et al., Additive Decoders for Latent Variables Identification and Cartesian-Product Extrapolation, in NeurIPS 23.

[3] Hyv¨arinen et al., Nonlinear ica using auxiliary variables and generalized contrastive learning, in AISTATS, 19.

[4] Khemakhem et al., Variational autoencoders and nonlinear ica: A unifying framework, in AISTATS, 20.

[5] Khemakhem et al., Ice-beem: Identifiable conditional energy-based deep models based on nonlinear ica, in NeurIPS 20.

[6] Brady et al., Provably Learning Object-Centric Representations, in ICML 2023.

[7] Wiedemer et al., Provable Compositional Generalization for Object-Centric Learning, in ICLR 2024.

[8] Lachappelle et al., Disentanglement via mechanism sparsity regularization: A new principle for nonlinear ICA, in Conference on Causal Learning and Reasoning, 2022

Q4. The scope of the evaluation is limited to toy settings which is somewhat outdated given the recent progress in generative modelling.

A4. We appreciate the valuable comments. Following the reviewer’s suggestion, we conducted additional experiments on CelebA-HQ for attribute disentanglement and MultiShapeNet (MSN) for object disentanglement, respectively. Experimental details and results are included in Appendix A.8.

For the CelebA-HQ dataset, we use the attribute-mixing strategy to disentangle attribute factors. To verify the disentanglement of the learned representations, we swap each latent vector one by one between two images and present the resulting composition images in Figure 6 and 7. In the third column of each figure, we observe that while the source images lack bangs, the swapped images successfully generate bangs while preserving other attributes. Similarly, in the fourth and fifth columns, the facial expressions (e.g., smile) and skin tones of the target images are effectively transferred to the source images. These qualitative results demonstrate that our attribute-mixing strategy is capable of disentangling attribute factors, even in complex datasets like CelebA-HQ.

We also validate our method on MSN dataset with object-wise manipulation and unsupervised segmentation. For the object-wise manipulation task, we encode pairs of images into $N=5$ object representations and exchange random object latents between the pairs to construct composite images. As shown in Figure 8, our method successfully performed object-level insertion and removal, demonstrating that each latent representation distinctly captures individual objects. This confirms that our approach effectively disentangles object representations within the latent space.

For the unsupervised segmentation task, we measure FG-ARI, mIoU, mBO on object masks following common practices in object-centric literature. As our method does not have a built-in mechanism to directly express group memberships between pixels, we additionally train Spatial Broadcast Decoder on the frozen latent representations to predict explicit object masks for each latent representation (please refer to A2 and appendix A.8 for details). The results are reported in Table 17 in Appendix A.8. Among the competitive slot-attention based baselines, our method achieves second-best performances across all of three metrics. The high segmentation scores of L2C are mainly due to its slot-attention-based regularization term (see Equation 8 in the L2C paper), which explicitly encourages the slot masks to align with object shapes. Excluding L2C, our method outperforms rests of the baselines (LSD, SLATE) across all metrics, despite not employing a spatial clustering mechanism like slot attention. These results demonstrate the effectiveness of our framework in disentangling object representations in a complex dataset.

Q5. What challenges do the authors anticipate in applying this model to real-world, complex datasets, and how might they address these?

A5. One of the primary challenges in applying our model to real-world, complex datasets is the need for a diffusion model that can reliably estimate the likelihood of complex composite images. Fortunately, this challenge can be addressed by leveraging off-the-shelf diffusion models trained on large-scale datasets (e.g., Stable Diffusion) or fine-tuning them on the target dataset. Another practical challenge is that real-world datasets often consist of highly diverse and complex images, which may result in slower convergence if a randomly initialized encoder is used. To address this, employing a pretrained encoder (e.g., DINO) as commonly done in recent object-centric approaches, would likely improve training efficiency and representation quality. Lastly, our current mixing strategy is designed to disentangle attributes or objects separately, but it does not yet support discovering factors of variation with more complex compositional structures, such as jointly disentangling attributes and objects or handling hierarchical relationships. To address such cases, proper mixing strategies should be investigated to capture such intricate compositional structures. For example, to discover factors of variation with hierarchical structures, the mixing strategy could be adapted to enforce hierarchy-specific constraints, such as allowing exchanges only between nodes at the same level in the hierarchy. Exploring and embedding diverse compositional structures through advanced mixing strategies is an essential direction for future research, and we will investigate this in our future work.

Q3. Experiments are done on rather simple on rather simple synthetic datasets.

A3. We appreciate the valuable comments. We conducted additional experiments on CelebA-HQ for attribute disentanglement and MultiShapeNet for object disentanglement, respectively. Experimental details and results are included in Appendix A.8.

For the CelebA-HQ dataset, we use the attribute-mixing strategy to disentangle attribute factors. To verify the disentanglement of the learned representations, we swap each latent vector one by one between two images and present the resulting composition images in Figure 6 and 7. In the third column of each figure, we observe that while the source images lack bangs, the swapped images successfully generate bangs while preserving other attributes. Similarly, in the fourth and fifth columns, the facial expressions (e.g., smile) and skin tones of the target images are effectively transferred to the source images. These qualitative results demonstrate that our attribute-mixing strategy is capable of disentangling attribute factors, even in complex datasets like CelebA-HQ.

We also validate our method on MSN dataset with object-wise manipulation and unsupervised segmentation. For the object-wise manipulation task, we encode pairs of images into $N=5$ object representations and exchange random object latents between the pairs to construct composite images. As shown in Figure 8, our method successfully performed object-level insertion and removal, demonstrating that each latent representation distinctly captures individual objects. This confirms that our approach effectively disentangles object representations within the latent space.

For the unsupervised segmentation task, we measure FG-ARI, mIoU, mBO on object masks following common practices in object-centric literature. As our method does not have a built-in mechanism to directly express group memberships between pixels, we additionally train Spatial Broadcast Decoder on the frozen latent representations to predict explicit object masks for each latent representation (please refer to A2 and appendix A.8 for details). The results are reported in Table 17 in Appendix A.8. Among the competitive slot-attention based baselines, our method achieves second-best performances across all of three metrics. The high segmentation scores of L2C are mainly due to its slot-attention-based regularization term (see Equation 8 in the L2C paper), which explicitly encourages the slot masks to align with object shapes. Excluding L2C, our method outperforms rests of the baselines (LSD, SLATE) across all metrics, despite not employing a spatial clustering mechanism like slot attention. These results demonstrate the effectiveness of our framework in disentangling object representations in a complex dataset.

Q4. How would this generalize to more complex datasets where the exact factors of disentanglement might not be known?

A4. We appreciate the reviewer’s thoughtful question. As discussed in our response to Q1, we believe that knowing the exact information of factors of variation is not strictly necessary for our approach. The latent representation would be learned according to the given mixing strategy, which guides specific compositional structure that the latent should satisfy. When there are an extremely large number of factors of variation in complex datasets, it would likely be infeasible to disentangle each exact factor. We hypothesize that the model would instead learn to group factors of variation into meaningful clusters. If the factors of variation exhibit complex compositional structures, such as hierarchical relationships, extending our current mixing strategies would be necessary. For instance, a hierarchical structure could be explicitly defined by allowing mixing only between nodes at the same level, which could enable the discovery of hierarchical factors of variation. Investigating mixing strategies for datasets with diverse and complex factors of variation is an important and promising direction for future research, and we aim to explore this in our future work.

We thank the reviewer for valuable comments. Below, we respond to each of the individual questions.

Q1. It seems like the approach is only usable if the practitioner already knows the underlying factors they want to disentangle, as the latent mixing strategies take this knowledge under account.

A1. We appreciate the valuable comment. While it is true that our current approach requires prior knowledge on target underlying factors to determine proper mixing strategy, we note that prior works also rely on such knowledge as they must select proper inductive bias (eg, either information-theoretic objectives for attributes or slot-attention encoder for objects). However, we believe that our mixing strategies would be still applicable even without exact information about underlying GT factors of variation in the data. The mixing strategy serves as a general framework for defining the compositional structure we expect our latents to follow, allowing the model to discover factors that align with a user-defined compositional structure. For instance, applying a fixed attribute mixing strategy encourages the model to disentangle globally consistent and combinable factors (e.g., global lighting or style) that are always present in the scene. Conversely, applying object mixing strategies to the same scenes helps the model disentangle dynamically occurring components within a scene, such as individual objects. In this way, the choice of mixing strategy flexibly determines the type of factors being disentangled.

Q2. The experiments show results for either object disentanglement or attribute disentanglement but no experiments for joint object and attribute disentanglement.

A2. We appreciate the insightful comment. We agree that our current experiments focus on either object or attribute disentanglement separately, without addressing joint disentanglement. This choice was made because our primary goal in this work was to demonstrate the capability of our novel inductive bias (i.e., compositional bias) to disentangle objects or attributes within a single framework. By presenting separate experiments for object and attribute disentanglement, we aimed to clearly highlight the effectiveness of our method in achieving comparable performance on both tasks. We agree that exploring joint object and attribute disentanglement is an important direction that aligns with the broader goals of our framework. Although it is beyond the scope of the current work, we consider it an important direction for future research.

We thank the reviewer for valuable comments. Below, we respond to each of the individual questions.

Q1. The impact of paper can be more by showing results on real world data

A1. We appreciate the valuable comments. Following the reviewer’s suggestion, we conducted additional experiments on CelebA-HQ for attribute disentanglement and MultiShapeNet for object disentanglement, respectively. Experimental details and results are included in Appendix A.8.

For the CelebA-HQ dataset, we use the attribute-mixing strategy to disentangle attribute factors. To verify the disentanglement of the learned representations, we swap each latent vector one by one between two images and present the resulting composition images in Figure 6 and 7 in Appendix A.8. In the third column of each figure, we observe that while the source images lack bangs, the swapped images successfully generate bangs while preserving other attributes. Similarly, in the fourth and fifth columns, the facial expressions (e.g., smile) and skin tones of the target images are effectively transferred to the source images. These qualitative results demonstrate that our attribute-mixing strategy is capable of disentangling attribute factors, even in complex datasets like CelebA-HQ.

We also validate our method on MSN dataset with object-wise manipulation and unsupervised segmentation. For the object-wise manipulation task, we encode pairs of images into $N=5$ object representations and exchange random object latents between the pairs to construct composite images. As shown in Figure 8, our method successfully performed object-level insertion and removal, demonstrating that each latent representation distinctly captures individual objects. This confirms that our approach effectively disentangles object representations within the latent space.

For the unsupervised segmentation task, we measure FG-ARI, mIoU, mBO on object masks following common practices in object-centric literature. As our method does not have a built-in mechanism to directly express group memberships between pixels, we additionally train Spatial Broadcast Decoder on the frozen latent representations to predict explicit object masks for each latent representation (please refer to A2 and appendix A.8 for details). The results are reported in Table 17 in Appendix A.8. Among the competitive slot-attention based baselines, our method achieves second-best performances across all of three metrics. The high segmentation scores of L2C are mainly due to its slot-attention-based regularization term (see Equation 8 in the L2C paper), which explicitly encourages the slot masks to align with object shapes. Excluding L2C, our method outperforms rests of the baselines (LSD, SLATE) across all metrics, despite not employing a spatial clustering mechanism like slot attention. These results demonstrate the effectiveness of our framework in disentangling object representations in a complex dataset.

Q2. Are there any further insights on the failure cases? Is it harder to compose attributes or objects?

A2. A failure case we observed occurs when a single latent encodes multiple factors of variation. However, this does not conflict with our compositional objective, as the objective remains satisfied even in such cases. We found that such issues can be mitigated by adjusting the weight of the compositional consistency loss. Specifically, the denominator of the compositional consistency loss includes a term that increases the distances between latents encoded from different images. This encourages the model to naturally encode distinct factors into separate latents, preventing the formation of empty or redundant latents in order to maximize the distance between each latent. In practice, we found that disentangling objects is a bit more challenging than disentangling attributes. This is due to spatial overlap between objects when they are randomly composed, which can occasionally result in unrealistic images. While this does not significantly impact overall training, it could cause slower convergence compared to attribute mixing, where mutually exclusive attributes are always composed without overlap.

We thank the reviewer for valuable comments and suggestions. We have revised the paper to correct the typos and provide clarifications. Below, we respond to each of the individual questions.

Q1. Complexity of the proposed approach leads to limited applicability and impact: the proposed approach requires the use of pretrained diffusion models to operate and requires access to composite images to train the model.

A1. We appreciate the reviewer’s comment and acknowledge the point about the need for pretrained diffusion models. Fortunately, with recent advancements in diffusion models, large-scale, off-the-shelf, pretrained models (e.g., Stable Diffusion) are now readily accessible. By leveraging these models, we can avoid the additional computational burden of training diffusion models from scratch when working with real-world datasets. Furthermore, generating composite images does not introduce additional learnable parameters, as we reuse the diffusion decoder trained with an auto-encoding loss. Therefore, while our method introduces some computational costs for generating and validating composite images, we do not need additional learnable components, preserving the overall applicability.

Q2. Limited performance increase in single-seed object disentanglement experiment.

A2. We appreciate the valuable comment. To examine the robustness of our method in object disentanglement, we trained our model using three different random seeds and report the standard deviations in Table 19 in Appendix A.9. Due to the limited time budget during the rebuttal period, we conducted this analysis only for our method but we will include standard deviations for all baseline methods as well in the final version of the paper to provide a comprehensive comparison. Regarding the limited performance increase in object disentanglement experiment, we would like to emphasize that the primary goal of our work is not to achieve state-of-the-art performance in both attribute and object disentanglement, but to propose a unified framework capable of disentangling both attributes and objects. While our method shows comparable performances to baselines in object disentanglement experiments, it is the only approach among the competitors that can successfully disentangles both attributes and objects within a single framework. We believe this novel capability of our method brings a meaningful contribution to the field.

Q3. Can authors elaborate on why the maximum likelihood is needed despite already enforcing low reconstruction error?

A3. We appreciate the reviewer’s comment. Minimizing the reconstruction error ensures that each latent representation is informative for a given image, but it does not inherently guarantee compositionality or realism in the generated composite images. To address this, we need an additional mechanism to encourage composite images to be realistic. Since ground-truth images for composite images are not available, we employ a pre-trained diffusion model to estimate the likelihood of composite images. By maximizing the likelihood, the model is guided to produce realistic composite images, thereby facilitating the learning of meaningful compositional representations.