Object centric architectures enable efficient causal representation learning

We would like to thank reviewer 9Arp for your valuable review, and we appreciate the time and effort you invested in evaluating our work.

Clarifications concerning the weaknesses

Weakness 1: We thank reviewer HmRq for bringing these works to our attention, however, we would like to emphasize that we are not claiming contribution to the sparse transitions in disentanglement, we are simply building on top of Ahuja et al. [1] as a modeling choice. Other choices are possible like the ones mentioned here by reviewer HmRq. We acknowledge that we should make this modeling choice more clear and mention the possibility of using the aforementioned works as well.

Weakness 2: We thank reviewer HmRq for bringing up this important point. While we acknowledge the usefulness of the additional insight provided by reporting per-property disentanglement scores, we would like to clarify that the presented scores aren't exactly average of per-property disentanglement scores. Intuitively, if say, we aim to disentangle $(p_x,p_y,\text{colour},\phi)$, and the model has only been able to separate just one property from the rest, then the score would be about $0.25$ or $\frac{1}{d}$, where $d$ is the number of target properties. If two properties are separated from the rest, then the score would be about $0.5$, and so on. So a score $0.95$ indicates that almost all properties are identified to a high degree. We hope this clarifies how the disentanglement scores work, and additionally, we will provide per-property disentanglement scores as well before the end of discussion period. Please expect another reply on this matter.

Answers to Questions:

1. Again, we thank the reviewer for bringing up this important observation. Above, we explained in more detail how these scores can be interpreted. However, we acknowledge that we have observed similar patterns to those that reviewer HmRq mentions. In particular, position properties are always far easier separated than the rest of the properties. We have also particularly witnessed more challenges with discrete properties, i.e., colour and shape, while rotation and size have also been fairly easy to disentangle. In summary, approximately we find the following ordering in increasing difficulty of separating the property:

$p_x,p_y < \text{size}, \text{rotation} < \text{colour}, \text{shape}$

However, we do believe there can be additional insight by reporting per property disentanglement scores, and we will provide those before the end of the discussion period.

We thank the reviewer again for raising these interesting points.

References

[1] Weakly supervised representation learning with sparse perturbations (Ahuja et al.)

We would like to thank reviewer T5fz for your valuable review, and we appreciate the time and effort you invested in evaluating our work, and we are glad that you have enjoyed reading our work.

Clarifications concerning the weaknesses

Weakness 2: We appreciate the reviewer's observation regarding our positioning and would like to address this concern. We acknowledge the need for a more explicit emphasis on the positioning of our work. While [1] explores a similar problem empirically, we want to highlight that our contribution lies in providing a provable disentanglement of object-centric representations, followed by strong experimental results against strong baselines. As outlined in our paper (please refer to the footnote on page 3), [1] does not address the identifiability of their approach, and our work fills this crucial gap by offering formally accompanied by empirical results. We thank the reviewer for drawing attention to this aspect and will make necessary revisions to emphasize this point more effectively.

Regarding slot matching in [2], we recognize the presence of some form of slot matching in their work, but it is essential to clarify that our primary contribution is not the matching algorithm itself. Instead, our focus is on identifying and addressing challenges that arise from objects violating standard assumptions for latent variable identification. We demonstrate how leveraging object-centric architectures can effectively handle these challenges, showcasing the sample efficiency of our method, as highlighted in the appendix. In summary, even though our matching approach differs from [2], it is not also a central contribution of our work. Moreover, [2] provides a benchmark for causal representation learning, and the matching used in [2] is for intervention modeling.

Weakness 1: In response to the concern raised, we would like to emphasize that our work addresses the previously unexplored territory of "provable" disentanglement of object-centric representations. Our approach connects identifiability results from prior work to real-world scenarios with multisets, filling a crucial gap in the literature. Furthermore, we believe the simplicity of our model adds to its strength. By demonstrating the effectiveness and consistency of a straightforward model across various settings, we aim to establish it as a reliable and strong approach for generalizing previous results to realistic environments.

Weakness 3:

We appreciate the reviewer's feedback on the proof sketch for Proposition 2. We would like to highlight that our goal has been to provide intuition without excessive mathematical formalism in this section, and for a more general proof that does not require identical objects, we refer the reader to see Hayes.

Answers to Questions:

1. Yes, empirical evidence supporting the claim in Sec. 4 about the object-centric model requiring $k$ times fewer perturbations is provided in Figure 7 in Appendix E.3. The observed benefits are not only quite significant but also surpass the $k$ times reduction, reinforcing the practical advantages of our proposed method in the finite sample regime.

2. The optimization problem in Eq. 13 for slot matching is computationally inexpensive, requiring only finding the minimum of a small array (same size as the number of objects). Moreover, as mentioned prior to Eq. 13, we do use the mentioned initialization trick and use slots obtained at $t$ to initialize those of $t+1$. However, this trick alone is not enough. We have observed that in simplistic 2D experiments with few objects and few target properties this trick works fine, however, in more complex settings, we observe that the order of extracted objects at $t,t+1$ is not quite preserved, and such occasional mismatch causes wrong training signals significantly preventing the optimization from finding disentangled solutions. Therefore, the use of this trick alone in more complex settings results in significantly suboptimal disentanglement, leaving some harder properties entangled altogether.

We would like to thank reviewer 9Arp for your valuable review, and we appreciate the time and effort you invested in evaluating our work.

Regarding the following,

Experiments are conducted with a single set of latent variables

we would like to clarify and highlight that our experiments are not carried out with only a single set of latent variables. While we deal with shared set of latents across objects, we show the effectiveness of our approach with varying set of latents, i.e., disentangling only $(p_x,p_y)$, or $(p_x,p_y,\text{colour},\text{shape})$, or $(p_x,p_y,\text{colour},\phi)$ (Please see the appendix for all the combinations we have tried.)

Clarifications concerning the weaknesses

Weakness 1: We would appreciate it if reviewer 9Arp could elaborate on this, and kindly let us understand what they have in mind, and what kinds of experiments would be interesting to be added to our work? We would like to highlight two aspects of this work in response to this concern. First, the experiments conducted in this work are significant steps toward realistic applications of causal representation learning (CRL), and they are significantly more challenging than the experiments of the current literature in causal representation learning that has mostly focused on very toy settings with one or two circles [1,2,3,4]. Secondly, an important aspect of evaluating such approaches in CRL is access to ground-truth which is usually achievable through synthetic datasets, yet we have tried to bridge the gap as much as possible by using synthetic samples that are realistic (CLEVR-style). The synthetic nature of our dataset provides extra handle on stress testing our method, and allows us to quickly identify the failure modes, necessary practical assumptions, etc. However, we strive to improve our work and are keen to learn more about the experiments that reviewer 9Arp believes would strengthen this work.

Weakness 2: In the presence of non-shared latents, the method still applies in its current form, however, unsurprisingly, we lose the benefit of requiring $k$ fewer perturbations. Good examples illustrating this point can be found in video games. There are properties such as characters moving around that are shared across all of them, and such properties are far easier to be identified due such knowledge being reused over and over. However special characters in the game might have superpowers unique to them, and consequently, learning about such special abilities is only possible through observing that one special character, without any knowledge transfer from the features of other characters who do not share this property.

Weakness 3: We thank reviewer 9Arp for the suggestion. We would like to highlight that despite the differences between the proposed dataset and our experiments at the surface level, these experiments are actually not qualitatively different in terms of disentanglement. The only difference is perception (i.e., slot attention + the preceding vision model), not the disentanglement approach. We know from other works using slot attention ([5,6,7,8]), that the perception works just fine in far more realistic settings than our shapes experiments and the proposed MNIST dataset. The proposed disentanglement method is contingent on good slot representations, and given the success of slot attention, we think there is limited qualitative difference in terms of disentanglement between our experiments and the reviewer's proposed dataset. [We are reading the question as meaning MNIST == realistic dataset, and asking us to create a dataset with multiple coloured MNIST digits with varying sizes, rotations etc.]

References

[1] Weakly Supervised Representation Learning with Sparse Perturbations (Ahuja et al.)

[2] Interventional Causal Representation Learning (Ahuja et al.)

[3] Partial Disentanglement via Mechanism Sparsity (Lachapelle et al.)

[4] Additive Decoders for Latent Variables Identification and Cartesian-Product Extrapolation (Lachapelle et al.)

[5] Bridging The Gap To Real-world Object-Centric Learning

[6] SlotFormer: Unsupervised Visual Dynamics Simulation With Object-centric Model

[7] Conditional Object-Centric Learning from Video

[8] SAVi++: Towards End-to-End Object-Centric Learning from Real-World Videos