Learning to Compose: Improving Object Centric Learning by Injecting Compositionality

We thank the reviewer for the positive comments. Below we respond to the individual questions.

Q1. Most experiments are conducted on synthetic objects, more experiments on realistic complex scenes, e.g., COCO, are desired.

A1. We appreciate the valuable comments. First, we would like to kindly note that learning object-centric representation in complex real-world images is widely perceived as challenging, and existing works rely on frozen pre-trained encoders and decoders to alleviate the problem. Since freezing the image encoder with a shallow slot attention module limits the opportunity to investigate the true impact of the compositional objective, we follow the standard settings that employ the synthetic data for evaluation.

Nonetheless, we agree with the reviewer that the synthetic scenes are insufficient to demonstrate the scalability of our compositional objective to complex scenes. Following the reviewer’s suggestion, we conducted preliminary experiments on the BDD100K dataset, which contains real-world driving scenes with many objects (cars, signs, buildings, trees, pedestrians, etc.) as well as complex interactions among objects and with the environment (occlusion, shadow, lighting, reflection, etc.). Following the prior work [1], we employed the pre-trained DINOv2 [2] as an image encoder and Stable Diffusion [3] as the generative prior. We then apply warm-up training for the one-shot decoder for 140K steps with an auto-encoding objective, and apply our compositional objective for the next 100K steps.

The results are summarized in Section B.7 in the Appendix (Figures 10 and 11). Based on visualization of the slot attention map (Figure 10), we observe that our method captures composable object instances not only from the foreground such as cars, buses, and trucks, but also from static environments such as road signs, trees, buildings, sidewalks, etc. In contrast, learning with only diffusion loss often separates an object into multiple slots, or contains multiple objects into a single slot. When composing two images by mixing their slots (Figure 11), we observe that our method successfully generates realistic scenes, modeling complex correlations among objects and environments. It appropriately adapts the appearance of newly added/removed objects, their shadow, reflections in the front glass and hood, and sometimes even global illumination change caused by removing the sun. Our method also generates more accurate compositions compared to the baseline, which often fails to adhere to the addition or removal of the objects.

To summarize, our preliminary experiments showed promising results that the proposed compositional objective can scale to complex scenes and model complex correlations among objects. We believe that incorporating the compositional objective in self-supervised learning can greatly enhance object-centric learning in complex scenes through end-to-end training with the deep encoder, which we plan to investigate in future work.

[1] Jiang et al., Object-centric slot diffusion, NeurIPS 2023.

[2] Oquab et al., DINOv2: Learning robust visual features without supervision, arXiv preprint 2023.

[3] Rombach et al., High resolution image synthesis with latent diffusion models, CVPR 2022.

Q2. Comparison to DINOSAUR.

A2. We appreciate the comment. As suggested by the reviewer, we included the DINOSAUR as an additional baseline and reported the comparison results in the table below. Although it performs reasonably well in CLEVRTex and PTR datasets, it fails dramatically in MutiShapeNet and Super-CLEVR datasets; we observe that it is because DINOSAUR encodes the entire foreground objects into a single slot in these datasets. Such failures are attributed to the frozen image encoder, which fails to generalize to domains unseen during the pre-training. It shows that learning the image encoder end-to-end with object-centric representation is important for building a robust and general object-centric learning framework.

We would also like to note that the main contribution of the DINOSAUR is employing the pre-trained features in object-centric learning, which is orthogonal to our contribution and hence can be combined together. In Section B.7 in the Appendix, we provided results on a real-world dataset, where we incorporated the compositional loss on top of the pre-trained DINO.

---

Q3. There are some recent works focusing on the binding ability of the slots to specific object types or properties, e.g., “ Improving Object-centric Learning with Query Optimization”. Is it possible to employ the binding ability to enhance the interpretability of slot mixing and further enhance the composition ability?

A3. We appreciate the insightful comment. Indeed, our work investigates a general problem of learning compositional representation and employs simple mixing strategies for generality. We believe that incorporating the binding ability of the slots can enhance the mixing strategy by reducing the search space for the valid slot combination, and improving the interpretability of the composite representation which is useful especially in manipulation tasks.

Q3. May I recommend a bar plot for Figure 3? The current plot based on colored Xs is quite hard to read. Moreover, a bar plot would allow to show the standard deviation of the results over multiple independent runs, which is important to strengthen the claims made in the text.

A3. We appreciate the comment. We will replace Figure 3 with a bar or box plot to present the standard deviations of each run more clearly.

---

Q4. In composition generation (Figure 4), please specify whether the outputs are cherry-picked. If so, please provide cases where all three methods reasonably work for fair comparison.

A4. Figure 4 illustrates the common failure cases of the baselines, which are caused by encoding an object into multiple slots (e.g., separating an object into multiple parts). For a more comprehensive understanding of the baselines, we also present additional results in Section B.5 (Figure 9) in the Appendix. In this figure, we present composite generation results similar to Figure 4 but for the examples where the baselines capture an object into a slot successfully. It shows that despite the reasonable slot attention map, the composite images produced by the baselines often exhibit unsatisfactory quality by distorting objects or altering their appearances due to the addition or removal of the objects. It shows that the object-centric representation learned in the baselines is generally not as compositional as our method.

We thank the reviewer for the positive comments. Below we respond to the individual questions.

Q1. In the proposed method, the number of losses, regularizers, and training tricks to balance is high. I would appreciate a more thorough ablation study where each component is isolated and tested. The combinations in table 2 are valid but do not cover all possibilities.

A1. We appreciate the valuable comment. Following the reviewer’s suggestion, we report the ablation study on all valid combinations of the components in the table below. We observe that the overall trends remain the same: considering the composition of two images for object-centric learning, either via composition loss or regularization, is helpful, and adding the slot initialization also improves the performance as well as enabling stable training and faster convergence. Overall, all components of our method contribute to the improvement, while employing composition loss and regularization are the most critical. We appreciate the comment and will include the results in the paper.

---

Q2. Can you provide non-segmentation based evaluation metrics? They can be the same as in Jiang et al. (2023) for example.

A2. We appreciate the valuable comment. Following the reviewer’s suggestion, we follow the evaluation protocol of Jiang et al. (2023) and evaluate the quality of the learned representation by predicting the object properties from the slots using object-wise annotations in CLEVRTex, PTR, and Super-CLEVR datasets (we could not include the MultiShapeNet dataset due to missing object labels). We report the results in Section B.6 in the Appendix and the tables below.

Overall, we observe that our method consistently outperforms the baselines in all datasets and properties, showing that the learned representations capture the characteristics of the objects well. We found that the prediction error in Super-CLEVR dataset tends to be higher than the other datasets, since it contains many small and occluded objects. We appreciate the comment and will include the results in the paper.

We thank the reviewer for the positive comments. Below we respond to the individual questions.

Q1. Suggestions to improve presentation

A1. We appreciate the valuable comment. We will modify Figure 1 and the first paragraph of the Section 3 to clarify that we are learning two decoders simultaneously.

---

Q2. Why background slots do not conflict: Could the authors explain whether / why, if not, background slots conflict when sampling slots with the random mixing strategy? In this case 1) is there any reason to expect canonical slot ordering, such that two background slots are unlikely to be sampled or 2) any reason why if two background slots are sampled, this does not lead to invalid scenes?

A2. We appreciate the comment. In the random mixing strategy, each slot is randomly and independently initialized in each image, and combined into a composite representation by randomly sampling slots from two images. Due to this randomness, there is a chance that mixed slots (i.e., composite representation) lead to invalid combinations, such as the ones composed only with the objects without background. To avoid this, we share the slot initialization between two images, and mix the slots exclusively according to their slot indices (e.g., we can sample i-th slot either from the first or second images, but NOT from both). Formally, let $I_1$ and $I_2$ be a random partition of slot indices i.e., $I_1\\cup I_2=\\{1,...,N\\}, I_1\\cap I_2=\\emptyset$. Then we construct the composite slot by $\\mathbf{S}^c=\\mathbf{S}^1_{I_1}\\cup \\mathbf{S}^2_{I_2}$, where $\\mathbf{S}^1$ and $\\mathbf{S}^2$ are slots extracted from the first and the second images, respectively. Since the slots are initialized identically between two images and the slots extracted from the identical initialization never appear twice in the composite representation, the model can learn to associate certain slot initialization to a specific part of a scene, such that the composite representation is ensured to be always valid. Indeed, we observed that the model learns to associate the background to a certain slot, as shown in Figure 6 in the appendix. We appreciate the comment and will revise the paper to clarify this point.

---

Q3. In ablation study (Table 2), why and how Regularization+Shared Slot Initializations is comparable to using only compositional prior?

A3. We would like to first note that both the regularization and compositional loss (generative prior) contribute to enhancing the compositionality of the representation since they are computed from the composition of two images and provide learning signals orthogonal to the auto-encoding path. We observe that the two losses also promote complementary behaviors to the model; The composition loss provides a direct learning signal that the composite image should be realistic, promoting the compositionality of the slot representation. However, we observed that it sometimes captures loose object boundaries especially when the background is monotonic, since they are highly correlated hence compositional. On the other hand, the regularization loss encourages the part of the image captured by a slot to be consistent before and after composition, without considering the realism of the composite image. Combining these two, our method learns to capture compositional components of a scene to a slot while maintaining the consistent association between the slot and a scene, leading to the highest performance.

Q5. Please clarify how the denoising diffusion and auto-encoding fit together in the proposed framework.

A5. Our diffusion decoder is trained with auto-encoding loss (Eq.(4) in the paper) while also used to compute the composition loss (Eq.(6)). In the auto-encoding path, the diffusion decoder learns the marginal distribution of data, while in the composition path, it is used to evaluate the likelihood of the composite image. We apply the stop gradient to the diffusion decoder in the composition path to ensure that it learns with only the true data distribution through the auto-encoding path. We will clarify the paper to make it more clear.

---

Q6. Why does the ARI performance on Super-CLEVR drop a lot compared to other datasets? Please provide an intuition on this.

A6. The ground-truth object masks of the Super-CLEVR dataset are much more fine-grained than the other datasets. Since the resolution of the attention map in our method and baselines is much coarser than the mask, we observe that it generally degrades the FG-ARI score for all baselines (e.g., a small error in a low-resolution attention map can lead to a bigger penalty), especially when the objects are occluded by the others (e.g., occluded by the spoke of a bike).

Q3. It would be good to also compare to a vanilla slot attention as a baseline.

A3. We appreciate the comment. As suggested by the reviewer, we additionally conducted experiments with vanilla slot attention. The results are summarized in the tables below. Overall, the performance of the vanilla slot attention is considerably lower than the other baselines and ours except in the PTR dataset where it achieves slightly better mIoU and mBO than ours. We observe that it is because the slot attention tends to capture tight object boundaries while missing some object parts (e.g., legs of tables), which is favored in mIoU and mBO but results in much lower FG-ARI scores. In all other datasets (CLEVRTex, MultiShapeNet, and Super-CLEVR), however, the vanilla slot attention fails to capture meaningful object masks, often suffering from position bias or leveraging simple color cues due to weak pixel mixture decoder as commonly reported in the literature. We appreciate the comment and will add the results to the main paper.

---

Q4. Please elaborate more on why sharing slot initialization helps avoiding bad mixing of objects?

A4. We appreciate the comment. In the random mixing strategy, each slot is randomly and independently initialized in each image, and combined into a composite representation by randomly sampling slots from two images. Due to this randomness, there is a chance that mixed slots (i.e., composite representation) lead to invalid combinations, such as the ones composed only with the objects without background. To avoid this, we share the slot initialization between two images, and mix the slots exclusively according to their slot indices (e.g., we can sample i-th slot either from the first or second images, but NOT from both). Formally, let $I_1$ and $I_2$ be a random partition of slot indices i.e., $I_1\cup I_2=\\{ 1,...,N\\}, I_1\cap I_2=\\emptyset$. Then we construct the composite slot by $\\mathbf{S}^c=\\mathbf{S}^1_{I_1}\\cup \\mathbf{S}^2_{I_2}$, where $\\mathbf{S}^1$ and $\\mathbf{S}^2$ are slots extracted from the first and the second images, respectively. Since the slots are initialized identically between two images and the slots extracted from the identical initialization never appear twice in the composite representation, the model can learn to associate certain slot initialization to a specific part of a scene, such that the composite representation is ensured to be always valid. Indeed, we observed that the model learns to associate the background to a certain slot, as shown in Figure 6 in the appendix. We appreciate the comment and will revise the paper to clarify this point.

We thank the reviewer for the positive comments. Below we respond to the individual questions.

Q1. "The experiments are performed over synthetic data only. I recommend exploring scalability to real-world data too.", "Please provide compositional generation results including interesting interactions among the objects".

A1. We appreciate the valuable comments. First, we would like to kindly note that learning object-centric representation in complex real-world images is widely perceived as challenging, and existing works rely on frozen pre-trained encoders and decoders to alleviate the problem. Since freezing the image encoder with a shallow slot attention module limits the opportunity to investigate the true impact of the compositional objective, we follow the standard settings that employ the synthetic data for evaluation.

Nonetheless, we agree with the reviewer that the synthetic scenes are insufficient to demonstrate the scalability of our compositional objective to complex scenes. Following the reviewer’s suggestion, we conducted preliminary experiments on the BDD100K dataset, which contains real-world driving scenes with many objects (cars, signs, buildings, trees, pedestrians, etc.) as well as complex interactions among objects and with the environment (occlusion, shadow, lighting, reflection, etc.). Following the prior work [1], we employed the pre-trained DINOv2 [2] as an image encoder and Stable Diffusion [3] as the generative prior. We then apply warm-up training for the one-shot decoder for 140K steps with an auto-encoding objective, and apply our compositional objective for the next 100K steps.

The results are summarized in Section B.7 in the Appendix (Figures 10 and 11). Based on visualization of the slot attention map (Figure 10), we observe that our method captures composable object instances not only from the foreground such as cars, buses, and trucks, but also from static environments such as road signs, trees, buildings, sidewalks, etc. In contrast, learning with only diffusion loss often separates an object into multiple slots, or contains multiple objects into a single slot. When composing two images by mixing their slots (Figure 11), we observe that our method successfully generates realistic scenes, modeling complex correlations among objects and environments. It appropriately adapts the appearance of newly added/removed objects, their shadow, reflections in the front glass and hood, and sometimes even global illumination change caused by removing the sun. Our method also generates more accurate compositions compared to the baseline, which often fails to adhere to the addition or removal of the objects.

To summarize, our preliminary experiments showed promising results that the proposed compositional objective can scale to complex scenes and model complex correlations among objects. We believe that incorporating the compositional objective in self-supervised learning can greatly enhance object-centric learning in complex scenes through end-to-end training with the deep encoder, which we plan to investigate in future work.

[1] Jiang et al., Object-centric slot diffusion, NeurIPS 2023.

[2] Oquab et al., DINOv2: Learning robust visual features without supervision, arXiv preprint 2023.

[3] Rombach et al., High resolution image synthesis with latent diffusion models, CVPR 2022.

---

Q2. It would be good to cite and discuss the “Object Discovery with a Copy-Pasting GAN” paper as a related prior work that also uses mixes over image to encourage object discovery.

A2. We appreciate the valuable comment on the related work. We agree with the reviewers that Copy-Pasting GAN (CP-GAN) shares similar motivation to ours in a spirit of discovering object masks by maximizing the realism of a copy-and-pasted image. Here we outline two major differences as follows. Firstly, our work focuses on learning compositional representation, which can be further employed in downstream tasks or compositional generation tasks, while the CP-GAN focuses on discovering object masks. Secondly, CP-GAN simulates the composition of objects by a simple copy-and-paste on RGB space, while our method combines two scenes using the object (slot) representation and employ powerful diffusion decoder to render the composite scene. We will include these discussions in the paper.