Slot-Guided Adaptation of Pre-trained Diffusion Models for Object-Centric Learning and Compositional Generation

We thank reviewer qbBW for their detailed comments and valuable feedback. The reviewer found our motivation for using adapters reasonable, and acknowledged the novelty of both our guidance loss and compositional generation ability, particularly noting our results on complex real-world images. They expressed concern about our language regarding reconstructions. We agree that there are some changes in the images (also noted by reviewers 8w5Y and Hqxz) and will revise this language in the updated version. We will also address their noted minor issues regarding language/grammar errors and missing text references for figures.

The reviewer raised four questions, which we address below:

1. Effects of adding guidance loss from the beginning of training: Our intuition was that initially, with random initialization, segmentation masks would lack useful supervisory information. Therefore, we chose to introduce guidance loss after the model learned useful image details. However, we conducted the suggested experiment. While training didn't diverge in this experiment, results were inferior to our original strategy of introducing guidance loss after 40K iterations:

2. Choice of SD 1.5 for COCO and SD 2.1 for VOC: In initial experiments, SD1.5 performed better on COCO while SD2.1 excelled on VOC. We attribute this to dataset size differences: COCO has approximately 118K images while VOC has 12K. Given SD2.1's enhanced capabilities and larger training dataset, we believe it adapted more easily to VOC's relatively limited dataset. Regarding v-prediction, we didn't observe the degeneration issues mentioned in SlotDiffusion's Appendix E.5, likely because we used pretrained models rather than training from scratch.

3. Classifier-free guidance (CFG) experimentation: We initially used a CFG value of 2.5 based on observed visual improvements. SlotDiffusion tried CFG but saw no improvements while LSD uses CFG value of 1.3. Following the reviewer's question, we conducted comprehensive CFG experiments with our best model on COCO dataset:

We will revise compositional generation and reconstruction results presented in the paper accordingly. Thanks for the suggestion.

4. Alternative register token approach:

We appreciate the reviewer's suggestion of using an additional slot in the slot attention module dedicated to capturing global information. We implemented the suggestion and our experiments showed mixed results:

While this approach improved several segmentation metrics, reconstruction performance slightly decreased. We hypothesize that the additional slot token's limited representation capacity (due to competitive attention in slot attention) prevents it from fully capturing global scene information, unlike our slot average token which incorporates information from all tokens. We will include these findings in the updated version.

I thank the authors for their reply. After reading other reviewers' comments, I will maintain my original rating of accept. I believe this work is an important milestone for the object-centric representation field.

We thank reviewer 8w5Y for their in-depth analysis and valuable feedback. The reviewer found our paper easy to follow and our contributions effective and novel. They raised several concerns, which we address below:

Regarding the evaluation of reconstruction performance and compositional generation ability: Following their suggestion, we evaluated all models for reconstruction performance on COCO using FID and KID scores, which measure the similarity between the generation distribution and ground truth data distribution. We also assessed the compositional generation ability of our model, in comparison to LSD and SlotDiffusion, by applying SlotDiffusion's compositional generation evaluation procedure, where slots in the batch are randomly mixed and generation is conditioned on these mixed slots (see SlotDiffusion paper Fig. 6; implementation from the official SlotDiffusion repository: https://github.com/Wuziyi616/SlotDiffusion/blob/main/slotdiffusion/img_based/test_comp_gen.py). Below are the results for both compositional generation and reconstruction performance:

Compositional Generation

Reconstructions

While we initially used CFG=2.5 for all generations, we discovered that CFG=1.3 performs better and include those results below (CFG parameter search was conducted following reviewer qbBW's question):

2. Our motivation for using guidance loss is to leverage the generative prior of pretrained diffusion models. This prior reflects learned internal representations of objects' identity, structure, and composition, derived from training on vast datasets. We anticipate this approach helps mitigate over and under-segmentation problems, as evidenced in Table 2 (Guidance ablations). While we don't see improvement in FG-ARI scores for object discovery, the improvements in mBO and mIoU metrics are substantial, indicating better spatial accuracy in segmentation results.

   We also verified that the results in Fig. 2 are not seed-dependent. Changing seeds produces the same outcome, with objects simply bound to different slots. We will include these results in the updated version.

3. We acknowledge the incorrect language regarding "perfect" reconstructions (also noted by reviewers qbBW and uBXM) and will revise it to "highly faithful to the original images." We evaluated all methods using FID and KID and the results show that our method outperforms all baselines in both reconstruction and compositional generation tasks. We will include these results and additional qualitative examples in the updated paper.

4. We appreciate the reviewer highlighting missing references, which we will add in the updated version.

5. The reviewer raised an important point about limitations. We agree that using a pre-trained diffusion model, primarily trained on real-world data, might limit performance on synthetic datasets like CLEVR/CLEVRTex. While facial information might be easier to adapt due to the presence of face data in the training set of diffusion models, synthetic datasets could pose greater challenges. We will discuss this limitation in the updated paper.

Regarding the reviewer's two questions:

1. Number of slots and usage of 80 slots: We determine the slot count by the maximum number of objects per image, following DINOSAUR's approach, which all baselines (LSD and Slot Diffusion) also adopt. Slots represent instances rather than classes, allowing different instances of the same class to occupy different slots. Nevertheless, following the reviewer's question, we tested using 80 slots (matching COCO dataset classes). This resulted in over-splitting and decreased performance due to the slot count far exceeding typical scene object counts. Training with 80 slots is indeed slower due to increased attention queries in slot attention (80 vs. 7) and adapter keys (80 vs. 7). Rather than using the class count for slot numbers, a better alternative would be dynamically setting slots per image - an active research topic [1] as we mention in our conclusions.

2. Results on MOVi-E dataset and experimental settings (Table 3): The MOVi-E results come from the LSD paper, with identical experimental settings except for our use of pretrained diffusion model and VAE.

[1] Fan, K., Bai, Z., Xiao, T., He, T., Horn, M., Fu, Y., ... & Zhang, Z. (2024). Adaptive slot attention: Object discovery with dynamic slot number. CVPR 2024.

We thank reviewer Hqxz for their detailed comments and valuable feedback..The reviewer highlights our introduction of adapter layers for slot-based conditioning as a key strength, emphasizing that it allows the model to move beyond text embedding space representations while maintaining the generative capabilities of pre-trained diffusion models. They also acknowledge our results for compositional image generation on complex real-world images such as COCO. Their concerns primarily focus on writing and grammar issues, as well as slight changes and artifacts in the reconstructed images. We will address all grammatical and syntax issues in the updated version of the paper. Regarding reconstruction quality, we note that inversion methods are generally approximate, and reconstructions obtained by pretrained diffusion models are inherently imperfect; indeed, exact inversion remains an active research topic in the field [1,2]. We have also evaluated our method's generation quality using FID and KID metrics (as suggested by reviewer 8w5Y) and compared them to the baselines (see the reply to the reviewer 8w5Y).

The reviewer raised two key questions. First, regarding contributions beyond the usage of adapters: our method also introduces register token and guidance loss, which significantly improve results. We present these improvements in Table 1 and Table 2 of the main paper, respectively, demonstrating that both guidance loss and register token enhance performance.

Second, regarding whether our method's performance relies on segmentation quality: our primary goal is to learn object-centric representations rather than merely performing segmentation. We then feed these object-centric representations to the diffusion model to guide the generation process. Our method can both learn representations specific to scene objects and perform object segmentation. We chose slot attention for learning object-centric representations, following LSD and SlotDiffusion. To explore potential improvements, we tested BOQ-SA [3], an enhanced version of slot attention which also yields better segmentation results as asked by the reviewer. The results show modest improvements across several metrics, as detailed below:

[1] Wallace, B., Gokul, A., & Naik, N. (2023). EDICT: Exact Diffusion Inversion via Coupled Transformations. In CVPR (pp. 22532-22541).

[2] Hong, S., Lee, K., Jeon, S. Y., Bae, H., & Chun, S. Y. (2024). On Exact Inversion of DPM-Solvers. In CVPR 2024.

[3] Jia, B., Liu, Y., & Huang, S. (2023). Improving object-centric learning with query optimization. ICLR 2023.

accidentally posted the same response twice, response deleted

We thank reviewer uBXM for their detailed comments and feedback. They highlighted our work's strong performance and noted that the paper is well-written and easy to follow. However, they expressed concerns about the method's novelty. While we acknowledge that adapters are widely used in diffusion literature, our work is the first to apply them to the object-centric learning problem with a distinct motivation: utilizing information from the pretrained diffusion model without disrupting its prior knowledge. Moreover, our method introduces two additional key contributions: the register token and guidance loss.

The reviewer's second point addressed the lack of ablation studies regarding the effects of guidance loss. We note that Table 2 in the main paper already presents these results, demonstrating the impact of different guidance loss types. The data shows that implementing guidance loss improves performance across all metrics except FG-ARI.