We thank the reviewer for their time and valuable feedback. We address each of your comments below:

Comment: “The discussion on the relationship between disentanglement and compositional generalisation could be improved by engaging with further existing work.”

Response:

Thank you for your comment. We have included a new paragraph, "On the Relationship Between Disentanglement and Compositional Generalization" discussing this point further in Appendix G.1. To summarize, our observation that disentanglement does not imply compositional generalization is theoretical in nature and is the same point made in [1, 2]. It amounts to saying that if a model inverts the ground-truth generator on the training set, i.e., achieves disentanglement, this does not imply mathematically that it also inverts the generator out-of-domain, i.e., achieves compositional generalization. Compositional generalization is only theoretically possible if a model’s decoder is enforced to have a certain structure on the full latent space. From an empirical perspective, however, it is possible that hidden inductive biases in a model could lead to a decoder having such a structure on the full latent space, which could potentially explain the results referenced by the reviewer, and would not be at odds with our theoretical observation

Comment: “The position of Montero seems somewhat incongruent with your argument that they both share an underlying mechanism of interaction asymmetry..”

Thank you for raising this nuanced point! The reviewer is correct in pointing out that our work states that the same properties on a model which enable disentanglement are also sufficient for enabling compositional generalization. We emphasize, however, that this is different from stating that disentanglement implies compositional generalization. Specifically, in Theorem 4.3, we show that if certain properties on a models decoder are satisfied in-domain on $\mathcal{Z}_{supp}$, then a model will be disentangled, while Theorem 4.4 states that if these same properties are enforced out-of-domain on all of $\mathcal{Z}$, then a model will generalize compositionally. Importantly, however, enforcing these properties in-domain does not imply that they will also be enforced out-of-domain. To enforce the properties out-of-domain requires additional methodological contributions, which we discuss in detail in Section H.3.

Comment:  “I am slightly confused by some terminology: does ‘latent’ mean latent dimension or latent vector?”

Response:

Thank you for raising this point! The reviewer is correct in noting that we overloaded the word latent in these context. We have updated the manuscript such that line 197 now reads “groups of latent components” while line 409 has been changed to “latent vector”.

Comment: “It seems like there are three notions here, the interaction asymmetry condition, 'sufficiently predictable' and compositional generalisation.”

Response:

Thank you for your comment. We apologize for the confusion and hope we can clarify this point. We first note that our use of “sufficiently predictable” is not meant to introduce a new mathematical notion. In Section 2, line 152, we discuss that for a model to generalize compositionally, one must be able to determine how a decoder $\boldsymbol{f}$ should behave out-of-domain on $\mathcal{Z}$, by only knowing its behavior in-domain on $\mathcal{Z}_{supp}$. In this sense, the behavior of the function must be “predictable” from its behavior in-domain. Hence, "predictable" is used in an intuitive fashion to describe this problem. To this end, Figure 2, aims to communicate that by restricting interactions via interaction asymmetry, the behavior of $\boldsymbol{f}$ out-of-domain on $\mathcal{Z}$ can be determined its behavior in-domain, because the cross partial derivatives are simple functions, i.e. polynomials. Thus the function is “predictable” in this sense. We hope this provides some clarity on this point, and based on your comment we have removed the phrase “sufficiently” to avoid potentially viewing this phrase as a new mathematical notion.

Comment: “The description of ARI on lines 478-9 is not the standard one as used”

Response:

In our experiments, we use the adjusted rand index from [3], which is a standard metric used in object-centric learning. We use the implementation from scikit-learn which can be found here. We hope this is sufficient to resolve the reviewers comment. Otherwise, we can add a more detailed description of the metric to our appendix.

Comment: “ did you try a range of values for $\beta$” ,.. it would be more insightful to report the exact values used.”

Response:

Thank you for this question! In our experiments on both Sprites and CLEVR6, we used values of 0.05 for both $\alpha$ and $\beta$. This was stated in line 2515 of our initial manuscript, however, we agree with the reviewer that adding these values to Table 1 would enhance clarity, and have thus made this addition to Table 1. Regarding, how these values were chosen, we did indeed explore different values for both parameters and found the stated values to work robustly. We did not perform extensive hyperparameter sweeps over $\beta$, however, thus we cannot rule out if certain values can be chosen such that using $\alpha > 0$ makes no difference. We believe our results suggest, however, that there is benefit to training with $\alpha > 0$. Specifically, as discussed on line 528 of our initial manuscript, the success of just training with $\beta > 0$ seems to be due to this loss implicitly minimizing $\mathcal{L}_{\text{interact}}$. Despite this implicit regularization, however, one does still achieve much lower values for this loss when training with $\alpha > 0$. We have updated Fig 5 in Appendix J.3, to reflect this point. To this end, we do still expect to see improvements in disentanglement when training with $\alpha > 0$.

Comment: Some information from Appendix I should be in the main paper, at least the number of dimensions in each slot.

Response: Thank you for this feedback! Accordingly, we will include information such as the slot dimension for the camera ready version of this work.

Comment: This formulation of the interaction principle is binary, … How easy would it be to extend your formulation to provide a score in [0,1], rather than {0,1}?

Response:

Thank you for your question! We think there may be some misunderstanding, which we hope to clarify. The reviewer is correct that whether an inference model achieves disentanglement can be quantified via a continuous value in [0,1]. However, this is measured as the disentanglement of a model w.r.t. a given a ground-truth model. This is different than asking whether or not a given ground-truth model satisfies the assumptions necessary for disentanglement. It is possible that a continuous score could be formulated to measure this, however, this will only be possible if one has access to the ground-truth generative process, which is generally not feasible. We note that this point is not unique to our work but will be true for any theoretical disentanglement result which formulates assumptions on the ground-truth generative process.

[1] Lachapelle et al., 2024, Additive Decoders for Latent Variables Identification and Cartesian Product Extrapolation

[2] Wiedmer et al., 2024, Provable Compositional Generalization for Object-Centric Learning

[3] Hubert et al., 1985, Comparing partitions.

Comment: “why did the authors choose to use learned queries instead of randomly sampled queries,”

Response: This decision was based on recent work in object-centric learning from [2], which showed performance gains when using learned queries in Slot Attention. We have now added an explanation for this choice and a citation to this work to the manuscript.

[1] Brady et al., 2023, Provably Learning Object-Centric Representations

[2] Biza et al., 2023, Invariant Slot Attention: Object Discovery with Slot-Centric Reference Frames

Comment: “the proposed approach requires two additional terms (KL term and the regularizer term) and it may introduces additional burden on finding proper hyper-parameters. It is hard to tell if these inductive biases are favorable than architectural bias in slot attention methods"

Response:

We agree with the reviewer that our method requires hyperparameter selection which could present computational overhead for certain datasets. We note, however, that in our experiments, we did not find extensive hyperparameter tuning to be necessary. We have added a discussion paragraph on this point in Section H.2 . Similar to in our previous response, however, we would like to emphasize that we do not view our method as being in competition to Slot Attention but instead again highlight that both methods offer different tradeoffs.

Comment: If my understanding is correct, the transformer decoder lacks a self-attention layer, …, potentially limiting scalability.

Response:

Thank you for this insightful comment! The reviewer is correct that our Transformer decoder only uses cross-attention layers. To this end, we would first like to note that using self-attention in a decoder is not scalable if the task is to reconstruct individual pixels, since self-attention must be computed on all pixel queries. Consequently, methods which leverage self-attention in the decoder, rely on reconstructing image patches opposed to pixels. In our experiments reconstructing image patches on CLEVRTEx, we found that only using cross-attention layers was sufficient for strong object-disentanglement. However, it is possible that leveraging self-attention could be advantageous in even more complex settings. In this case, it is not immediately obvious how to adapt our attention regularizer since self-attention will introduce further interactions between slots. We have added a new discussion paragraph, "Self-Attention in Transformer Decoders.", discussing these points in more detail in Section H.2.

Comment: “The proposed J-ARI metric may be misleading. To assess whether composable abstractions are effectively captured, a downstream task like property prediction, may be necessary.”

Response:

Thank you for this nuanced comment! The reviewer is correct in pointing out this potential issue with our metrics which we did not previously discuss. We have added a new discussion paragraph “Latent Prediction-Based Disentanglement Metrics“ to Section H.2, discussing this point in detail. To briefly summarize, we agree with the reviewer that latent prediction metrics may be necessary to resolve the ambiguity of whether a given slot actually encodes multiple objects. With this being said, we argue that the metrics used in the works referenced by the reviewer, are insufficient for determining this. Specifically, these metrics indicate if an inferred slot contains all information about a given object, however, they do not indicate if an inferred slot contains information about more than one object. This issue was pointed out by [1] who aimed to address it by fitting an additional predictor between the second most informative ground-truth slot for a given inferred slot. In early experiments, we found the metric from [1] to yield inconsistent results on CLEVR6, which lead us to focus on decoder-based metrics. We leave it for future work to formulate a latent prediction metric which overcomes the aforementioned issues of prior works.

Comment: “can the trade-off between interaction and model expressivity be effectively controlled through the parameter “

Response:

We conjecture that it is possible to have more fine grained control of this trade-off through $\alpha$. While we did not observe a case in our runs on CLEVR6 in which our method did not model interactions, we were also not explicitly targeting this behavior. We note that inducing this behaviour would likely require carefully warming up the value of $\alpha$ throughout training since if the parameter is chosen to be too large at the beginning of training, we observed that models are often pushed towards sub-optimal solutions.

Comment: “wouldn’t a spatial broadcast decoder … represent the upper bound performance? Why does the proposed method outperform this approach?”

Response:

We hypothesize that the worse performance of Spatial Broadcast Decoders (SBDs) relative to our model on Sprites is due to the following reasons: Firstly, the alpha-blending layer in SBDs can indeed introduce interactions, such that these models do not give an upper bound on performance. Additionally, while occlusions are relatively rare on Sprites, they can occur, and are often more pronounced than on CLEVR6. Consequently, it is possible that the restrictions on interactions in SBDs can prevent the model from properly modelling this data, while a more flexible Transformer decoder can more easily model it.

We thank the reviewer for their time, and appreciate their comments highlighting the value of our theoretical contribution. We are also grateful for the insightful feedback on our empirical contribution. These comments lead us to address several nuances, which we believe has improved the quality and clarity of our work.

Before addressing your comments, we would first like to clarify our view on the contributions of this work. We view our theory as our core contribution. Such a theoretical foundation for both disentanglement and compositional generalization has thus far been lacking, and we view our main contribution as providing this foundation through a general principle, which also unifies and extends prior work.

Our main objective empirically was to showcase the practical utility of this theory in informing the design of novel learning methods. To this end, our aim is not to present our method as necessarily superior to existing methods. Instead, we aim to highlight that insights from our theory lead to a new object-centric learning method which yields strong object disentanglement while potentially offering different advantages in terms of scalability compared to existing methods.

We now address each of your comments below:

Comment: “the experimental support is relatively limited, …raising questions about the generalizability of the findings to more complex scenarios.”

Response:

We agree with the reviewer that adding additional experiments is important for understanding the generalizability of our method to more complex scenarios. We have thus conducted additional experiments on the CLEVRTex dataset which is a significant step up in complexity compared to CLEVR6. We found that our method is also able to achieve strong object disentanglement on this data, outperforming the baseline methods we tested against in terms of our metrics. We discuss these results in more detail in the general reply to all reviewers and have added a new section in the paper (Appendix I) presenting our experimental setup and results.

Comment: “minimizing interaction without knowing the ground truth value of “. “If the Sprite dataset exhibits 0th order interaction, a stronger regularizer may be necessary compared to that in CLEVR6.”

Response:

We agree that minimizing interactions without knowing n could potentially pose challenges empirically. In our experiments on Sprites and CLEVR6, however, we found the same hyperparamter values ($\alpha=0.05, \beta=0.05$) to work well across both datasets, though the weight on the reconstruction loss was changed from 5. to 1. on CLEVR6. We have now included a new discussion paragraph on hyperparameter selection in Section H.2.

Comment: “Clarifying the unique benefits of the proposed regularizer compared to slot attention’s inherent mechanism would enhance the paper's contributions.”

Response:

We thank the reviewer for their comment. We would like to emphasize that our aim is not to present our method as inherently superior to Slot Attention, but instead we highlight that both methods offer different tradeoffs. For example, training with our loss enables using a general Transformer encoder, which potentially yields better scalability than encoders with more explicit object-centric priors such as Slot Attention. This comes at the cost of training with regularizers, however, which require hyperparameter selection. While our experiments did not require extensive hyperparameter tuning, it is possible that certain datasets will exhibit increased hyperparameter sensitivity. We have added a new discussion paragraph “Trade-offs with Slot Attention” in Section H.2, discussing these points along with additional details.

We also note, however, that, while the mechanism in Slot Attention provides a useful inductive bias for minimizing interactions between slots, it does not explicitly regularize for interactions, as in our method. Furthermore, we find that on all datasets we consider, our method yields slightly stronger object disentanglement compared to a Slot Attention-based Transformer autoencoder. This can also be seen visually by inspecting Jacobian plots for each dataset, which we have now added in Figures 6 and 7 in Appendix J.3.

We thank the reviewer for their valuable feedback. We also appreciate the positive assessment of our theoretical and empirical contribution as well the comments on the overall significance of our contribution.

We address each of your comments in detail below:

Comment: “The empirical evaluation, [...], is somewhat limited in scope.”

Response:

We agree with the reviewer that adding experiments on more complex data is important for understanding the practical utility and scalability of our model. We have thus conducted additional experiments on the CLEVRTex dataset which is a significant step up in complexity compared to CLEVR6. We found that our method is also able to achieve strong object disentanglement on this data, outperforming the baseline methods we tested against in terms of our metrics. We discuss these results in more detail in the general reply to all reviewers and have added a new section in the paper (Appendix I) presenting our experimental setup and results.

Comment: “It would be beneficial to discuss how these assumptions could impact the applicability of the model and whether there are potential relaxations that could make the theory more practical.”

Response:

Thank you for raising this point! We agree with the reviewer that additional discussion of the practical implications of our theoretical assumptions and whether relaxations are possible are important for the paper. We have thus included a new section (Section H.1), which includes several new discussion paragraphs to this end. To summarize, we have added discussion paragraphs on:

• A limitation of interaction asymmetry in that it requires the order of interaction to be the same for all slots. We discuss situations where this will likely not hold in practice and whether this assumption could potentially be relaxed.

• The implications of sufficient independence on the observed dimensionality of the data and whether this will hold in practice.

• Real-word concepts which might not be captured by interaction asymmetry.

• The restrictiveness of the aligned connectedness condition on the latent support from a mathematical standpoint.

Comment: “How does the interaction asymmetry principle relate to the Information Bottleneck theory?”

Response:

Thank you for this comment as well as the references. Based on your suggestion, we have included a discussion paragraph on the relationship between interaction asymmetry and the Information Bottleneck (IB) principle which can be found in Section H.1. To briefly summarize its contents, the principles have a key difference in that the IB principle aims to enforce conditions on the latent distribution, while interaction asymmetry is formalized only on the generator function, and thus does not impose any conditions on the latent distribution. With this being said, our theoretical results do yield insights which resemble the IB principle. Specifically, our results suggest that a model should encode observations $\boldsymbol{x}$ such that unnecessary latent dimensions contain no information about $\boldsymbol{x}$. This resembles the idea in the IB principle of minimizing mutual information between observations and latents.

We thank the reviewer for their valuable feedback and their positive assessment of our work. We address each of your comments below:

Comment: “Exploring the effectiveness of the proposed method on more complex [...] datasets […] is crucial for demonstrating its broader applicability."

Response:

We agree with the reviewer that adding experiments on more complex data is important to demonstrate our method’s broader applicability. We have thus conducted additional experiments on the CLEVRTex dataset which is a significant step up in complexity compared to CLEVR6. We found that our method is also able to achieve strong object disentanglement on this data, outperforming the baseline methods we tested against in terms of our metrics. We discuss these results in more detail in the general reply to all reviewers and have added a new section in the paper (Appendix I) presenting our experimental setup and results.

Comment: “Aligned connectedness seems restrictive and its practical implications not fully explored”.

Response:

Thank you for raising this point. We have now included a discussion paragraph in Section H.1 discussing the restrictiveness of the aligned connectedness condition. To briefly summarize, we discuss the restrictiveness of this assumption from a mathematical point of view, highlighting that it is not an overly restrictive or unrealistic condition. For example, any set that is convex is also aligned connected. We agree with the reviewer, however, that, in practice, understanding whether aligned-connectedness will generally hold, and to what extent it can be violated are important questions which we leave for future work.

Comment: . “Analyzing the gap between approximation and theoretical ideal, and exploring alternative regularization strategies that more directly address higher-order interactions, could enhance the method's effectiveness”

Response:

We agree with the reviewer that understanding the gap between our regularizer and the theoretical ideal is an interesting question. To this end, we computed the Pearson correlation between the value of our regularizer throughout training and the Jacobian Interaction Score, which can be seen as an exact measure of $0^{\text{th}}$ order interaction. On Sprites, we found an absolute correlation of .93, and .77 on CLEVR6. This indicates that while there is some gap between our regularizer and the theoretical ideal, the two scores are strongly correlated. Regarding directly regularizing higher-order interactions, as noted in Section F.2 of our paper, we agree with the reviewer that this is an important question, which we leave for future work to explore further.