Grounded Object-Centric Learning

We thank the reviewer for their detailed comments and constructive feedback. We very much appreciate the fact that our framework was received as logical, interesting, and insightful for the community.

I would appreciate more ablation studies in the experiments section. The current version primarily presents the state-of-the-art (SOTA) performance for two case studies, but additional ablation studies would provide further insights into the specific contributions and the impact of different components or techniques employed in the proposed method.

We agree that the ablation can provide more insights about the method; we have included ablations on sampling, types of spectral decomposition, and sample efficiency on multiple tasks and datasets. Additionally, we have included the ablations on codebook size and its implications on results in the appendix.
In case we are missing any ablations that would add to the clarity in the context of CoSA, please let us know.

The author mentions that the method incorporates the object-level property vector, but there is a lack of evidence regarding how it functions. For instance, it is unclear whether the method can effectively discriminate between multiple instances with similar appearances

The evidence of selecting object-level property vectors can be observed in both object discovery and reasoning tasks. In the case of object discovery, FG-ARI measures this property, whereas HMC measures the same for reasoning tasks measures this property, whereas for reasoning tasks, HMC measures the same.

Similarly, in the case of Object discovery, we have added new results to qualitatively demonstrate the binding across multiple instances; please refer to Figure 19 in the appendix section.

The visualization of the COCO results does not appear to be accurate, and it seems that the method may not be as applicable to real-world scenarios.

We do observe that the results on the COCO dataset are not as good as supervised segmentation models, but it is important to note that the proposed model is fully unsupervised, and based on the initial number of slots hyperparameter, the model does segregate between different types of objects in the image. We leave further exploration in improving the model performance to match closely to the segmentation ground truths to future work.

Besides performance gains, what evidence does the paper provide to show that the object-level property vectors are working effectively?

We demonstrate the notion of binding and qualitatively as well as quantitively. Qualitatively, this can be seen in the grounded dictionary visualization section in the main paper (Figure 3) and in appendix section K.1 and section H.5. Quantitively, the results are reflected in terms of FG-ARI and HMC metrics.

We thank the reviewer for their detailed comments and constructive feedback. We very much appreciate the fact that our work was found to be well-motivated, novel, and the illustrations were found interesting and interpretable.

The experiments on more complex scenes are required. For example, multiple instances of the same object category, it would be interesting to show the property binding ability in this case.

We share the reviewers view on examining the binding ability of the framework on multiple object instances of the same object. Given that CLEVR has several of these examples in the dataset, we included a section in Appendix H.5, where we demonstrate the reusability of dictionary elements based on repeated object instances. As illustrated in Figure 19, similar looking object slot representations are sampled from the same distribution.

Does the model build on pre-trained backbones? Or a randomly initialized encoder is also sufficient to provide cues for spectral decomposition and discritization?

For CLEVR, ObjectsRoom, Bitmoji, FFHQ, and Tetrominoes datasets we start with a randomly initialized encoder. For COCO, given the very high diversity of image content, we start with a pre-trained image encoder similar to DINOSAUR and apply CoSA on the extracted features.

I suggest authors to run the trained conditional slot attention with grounded slot dictionary on some video data, e.g., DAVIS-2017, to validate whether the slot dictionary can track objects across time and more vividly show the binding ability to object types.

We thank the reviewer for the great suggestion. We agree that extending the proposed method to a large-scale video dataset like DAVIS-2017 would be interesting and valuable. However, at this stage, we believe that this would be best served by a dedicated investigation in future work.

Nonetheless, to reassure the review of our method’s ability to correctly identify and bind multiple object instances, we have added a discussion section and some illustrations using CLEVR in the appendix (section H.5).

The conventional slot attention based methods use reconstruction loss as the self-supervised objective to guide object decomposition. What is the relationship between the objectives used in this work and the reconstruction loss?

The objective function in our work can be viewed as the sum of reconstruction loss (likelihood loss) + KL regularization. The additional KL term is used to train the strategies of selecting the elements from GSD for a given image.

Our work introduces a probabilistic perspective of slot attention by optimizing a principled variational lower bound of the log likelihood of the data. The image likelihood we maximize is related to the reconstruction loss in previous works in the sense that minimzing a MSE loss is analogous to maximizing a pixel-wise factored Gaussian likelihood.

Thank you for your response and feedback.

Based on your suggestion, we have included inference results on a video from the DAVIS dataset. Please refer to Figure 20 in the appendix, even in this case, we can observe that the CoSA consistently binds foreground and background slots to a particular GSD element.

We thank the reviewer for their detailed comments and constructive feedback. We very much appreciate the generally positive outlook and the fact that our work was found to be well-motivated and novel.

How much of the improved performance is from just predicting the number of slots instead of using a fixed K? What if we use the predicted number of slots without the GSD?

Our apologies for the confusion. Although it is technically possible to perform dynamic estimation of the number of slots K at training time using our method, it requires masking of surplus slots, which somewhat complicates the training pipeline without providing guaranteed benefits. Our method was used at inference time only; we have made this clear in the revised version of the paper.

Therefore, comparing performance improvements from training with the dynamic estimation of K versus the GSD is not applicable, since the GSD was always trained with fixed K. If we remove the GSD, we end up with the standard Slot-Attention model, which we compare with in our ablation study (e.g. see the SA model in Table 1, Table 2 etc).

What is the distribution of the number of objects for the different datasets? This would be important to interpreting the MAE values.

Thanks for highlighting this point, we have updated the dataset information in appendix section C, with the distribution of the number of objects for the different datasets.

While I can appreciate the probabilistic interpretation of the model, I feel it does not add to the clarity of the paper and may be more appropriate for the appendix. Specifically, if I understand correctly, if q(z|x) is deterministic and p(z) is uniform as in VQ-VAE, then the KL term is just a constant and not actually used to optimize the model?

We thank the reviewer for the astute observation. It is true that under a uniform prior and deterministic sampling, the KL term drops to a constant. However, this is not the case for the Gumbel sampling scenario, as while the prior is still uniform, the sampling is stochastic. For this reason, the probabilistic exposition in our paper is designed to be generally consistent with any (discrete) representation learning setup.

We have clarified this in the paper and have included additional background on quantization clarifying this in the appendix.

The discussion of the different sampling strategies (Euclidean, Cosine, Gumbel) does not seem necessary for the main text since (from my understanding) the experiments in the main text are only done with the Cosine version?

We thank the reviewer for this suggestion; we have briefly introduced different types of sampling for the sake of completion, providing different options available for constructing the dictionary. We have included details of quantization and the sampling methods in the appendix.

The codebook size seems like a potentially important parameter. What size do you choose for the different experiments? How sensitive are the results to codebook size?

In all our experiments, we initialize the codebook with a sufficiently large number of embeddings for the complexity of the datasets used. We’ve included additional ablations and further discussion on codebook size in the appendix section H.4.

Here are the ablation results on the CLEVR dataset.

I want to confirm that the Abstraction Module is not differentiable and just uses the output of the encoder, which is trained through the Slot Attention path. Then, there are similarly no gradients flowing through the GSD and it is only updated with EMA (Appendix F). Is this understanding correct?

Yes, the reviewer is indeed correct; the abstraction module is nonparametric, but the gradients flow through it.

In the case of GSD, the key part of the dictionary $\mathfrak{S}^1$ is updated via EMA, while the distribution parameters in $\mathfrak{S}^2$ are updated with gradients from the SA module.

How are the dynamic number of slots actually implemented during training?

Dynamic slot selection during training requires masking of surplus slots, which complicates the training pipeline without providing guaranteed benefits. For this reason, we trained with a fixed number of slots K, and apply dynamic slot selection at inference time only.