Explicitly Disentangled Representations in Object-Centric Learning

Thank you for reviewing our work and providing helpful feedback.

• Concerning the first weakness point, we think that explicitly designing the architecture for shape and texture disentanglement does not prevent it from being extended to other latent dimensions. For example, by employing the Invariant Slot Attention mechanism instead of the base Slot Attention mechanism (as we did in our work), it is possible to extend the latent dimensions to scale and position components. It is also feasible to do the same for rotation features by including the rotation factors in the ISA mechanism. Moreover, we think that texture is indeed one of the main properties describing an object, along with others such as position, size, shape, and orientation. Furthermore, assuming to have a strategy to extract additional object information, it would be possible to concatenate it to the texture and shape representations before the decoding phase to exploit it.

• For the second weakness point, we are not sure we understand the meaning of it. In fact, all SA-based models use both shape and texture information to decode the object textures: usually, they feed the complete slot vectors (encoding all the object information, texture and shape included) to the decoder. Could we please ask for a clarification on it? Thank you very much in advance.

• Regarding the last weakness pointed out, the Sobel filter is indeed responsible for part of the benefits of our approach. To analyze this, we decided to provide an ablation (Appendix F.2) on its impact in disentangling shape and texture. However, we do not consider this point to be a weakness, but a modeling choice. It would also be possible to replace the Sobel filter with a different filter, for instance, a more sophisticated one that could lead to improved performance.

• To answer the first question, we did not use the rotation factors in DISA as, on Tetrominos and CLEVR, ISA performed better without it. We extended this decision to the other datasets as well.

• Finally, we address your last question by including a qualitative analysis of the position and scale disentanglement (Appendix F.1 Figure 13). The results suggest that this property is indeed successfully achieved. Regarding the extent to which position and scale were varied in the experiments: we trained on the complete training sets, where the objects can be found in approximately every location, while the scales are dataset-dependent. On Tetrominoes, a single fixed size is present, while Multi-dSprites has 6 values linearly spaced in [0.5, 1], where 1 represents a given default object size. CLEVR has 2 sizes (small and large) while CLEVRTex has 3 (small, medium, and large). However, for these two datasets, the proximity of an object to the camera affects its size in the image.

We hope to have clarified all your doubts and addressed your concerns.

Thank you for the clarification.

We understand your concern expressed in "making a network invariant to a new factor X is an ad-hoc process which requires explicit architecture which is tailored for X...". However, we would like to respectfully point out that by following this argument to reject a paper, prior valuable contributions (such as [1][2][3]) could have been dismissed as well. In our opinion, employing these inductive biases to explicitly disentangle certain object properties does not imply that all other properties should be disentangled following the same (explicit) approach. For instance, to disentangle the sub-properties of a texture (color, material, ...) without assuming to know them a priori, an interesting direction would be to couple DISA with some hypothetical method aiming to separate generic factors of variation (e.g., beta-VAE in the context of probabilistic models). Therefore, we think this work can still provide a valuable contribution toward more interpretable, structured, and robust representations, without precluding the possibility of combining it with non-explicit disentanglement methods targeting different properties.

[1] Disentangling 3D Prototypical Networks For Few-Shot Concept Learning, https://arxiv.org/abs/2011.03367

[2] Unsupervised Part-Based Disentangling of Object Shape and Appearance, https://arxiv.org/abs/1903.06946

[3] Invariant Slot Attention: Object Discovery with Slot-Centric Reference Frames, https://arxiv.org/abs/2302.04973

Thank you for your helpful review and for raising relevant concerns.

• As for the first question, we agree that a study regarding the impact of the Sobel filter on the disentanglement of texture and shape is valuable. Therefore, we included it in Appendix F.2. In that section, we also similarly analyze the importance of the variance regularization on the disentanglement. Since we also concur with the importance of experimenting with a dataset with more complex textures, we trained on CLEVRTex and presented the results in the updated version of the paper.

• To answer the second question, we included in Appendix F.4 Figure 34 shape generation results. It is important to know that this capability is strongly limited by the nature of the datasets we trained on, as all of them present very little variability in terms of object shapes.

• Regarding the last question, basing DISA on ISA instead of SA is not strictly necessary. When using SA instead of ISA, position and scale information get included in texture and/or shape components, without preventing a quantitative study of the texture and shape disentanglement. However, there are two main reasons for choosing to employ ISA. (1) It enables a more efficient training process and (2) allows disentangling position and scale. In particular, the latter helps conduct the compositional and generative experiments. Without this property, during the texture transfer experiments, we may end up transferring between objects even position and scale information (which is undesired). Similarly, we may alter the position and scale of objects while generating new shapes.

We hope to have thoroughly addressed your concerns.

Thank you very much for the positive response to the revised paper.

CLEVRTex is indeed still difficult to tackle, with both DISA and its baselines. We are confident that future works will be able to address the remaining problems and obtain precise masks, reconstructions, and disentanglement even on such complex datasets.

Please let us know if you have any additional questions or concerns, as we would be happy to answer.

Thank you again for your time.

Thank you for your review and for raising valuable concerns.

• We worked to address the weaknesses you identified. The new version of the paper now includes an additional section (Appendix C) in which we compare in more detail two approaches (already mentioned in the related work) with DISA. Specifically, you can find a more exhaustive discussion on Lorenz et al. [1] in the second paragraph.

• We further included CLEVRTex in our experiments to evaluate the performances of DISA in a more complex and realistic setting. As reported in the updated version of the paper, DISA outperforms SA and ISA on object discovery (BG and FG ARI) and has very similar MSE to ISA. Since we detect objects solely on the filtered image while ISA does so on the original one, the ARI score results suggest that, even on CLEVRTex, the Sobel filter does not pose a limit in this regard.

• Concerning your first question, there are some limitations to the Sobel filter, such as the production of thick edges. Another limitation is that, as it is designed to approximate horizontal and vertical edges, it can be inaccurate with diagonal ones (e.g., very high gradients). A more sophisticated filter could, of course, be beneficial to DISA. In Appendix A, you can find an explanation of the Sobel filter, including a visual example (Figure 6) of how the Sobel filter works on all the considered datasets (CLEVRTex included, to show it on high-frequency texture patterns).

• Regarding your question about the variance regularization, we compute the variance across all the objects in the batch images. Precisely, for each slot component, we calculate its variance across all the slots in the batch. This is mentioned below Equation 7 with the sentence "When the loss is computed over a batch of images, $N_s$ becomes $N_s$ times the number of images in the batch".

We hope we addressed all your concerns.

[1] Unsupervised Part-Based Disentangling of Object Shape and Appearance, https://arxiv.org/pdf/1903.06946.pdf

Thank you for your insightful review and for raising fair concerns.

• Regarding the first question and weakness, we indeed think that generalization and downstream transfer are the principal motivations for investigating this direction. However, we do not study the benefits of using the representations learned by our model on downstream tasks as we think it would require a separate paper. In fact, to answer such a research question, we cannot rely on a single experiment such as the property prediction used in our work. We think it would be necessary to conduct an extensive study on more diverse tasks than simply property prediction, including for instance visual question answering and RL-related problems. In our opinion, this study should not be simply included in the appendix of a paper but be the core of a distinct one.

• To address the second question, we included a dedicated section in the appendix (Appendix A). Here, we present the math and intuitive idea behind the Sobel filter, other than examples of its application to all the considered datasets. We also included the more real-world dataset CLEVRTex, as we additionally trained on it and presented the results in the updated version of the paper.

• Concerning the last question and weakness pointed out, we indeed missed D3DP, which tackles the explicit disentanglement of shape and texture in an object-centric manner. However, they rely on ground-truth bounding boxes to decompose a scene into objects. Our method, instead, is completely unsupervised. Therefore, we changed that sentence to "However, to the best of our knowledge, no research has been carried out on the explicit disentanglement of the texture and shape dimensions in unsupervised object-centric learning, especially with non-probabilistic models.". Moreover, we included an additional section (Appendix C) in which, in the first paragraph, we compare DISA with D3DP. Finally, as for the comparison with AdaIn, we agree that it would be interesting to conduct an experiment using it when decoding textures. However, we unfortunately decided not to pursue it in favor of using our reserved computational resources for experimenting on CLEVRTex (i.e. extremely resource-consuming) and providing an ablatation study on the importance of variance regularization and Sobel filter for the disentanglement.

We hope we were clear and exhaustive with our answers and additional content in the paper.

Dear Reviewer Acn7,

Thank you again for your constructive comment.

As we are very close to the end of the discussion period, we would like to ask if all the concerns have been addressed and, in that case, to kindly remind you to adjust the rating accordingly (if needed). Otherwise, if you have additional questions, we are happy to answer.

Thank you very much in advance.

I went through the rebuttal, however I think in the current form, the paper is not ready for acceptance.

I think the paper lacks (as i have mentioned in my initial response):

(i) Analysis on the Scalability of Sobel Filter (comparision to other ways of disentangling shape from texture eg: ADAIN)

ii) Downstream comparision wrt other object-centric learning methods such as SloT Attention and ISA

Without these experiments the usefulness of the proposed method is unclear to me. The authors in the rebuttal think the comparision with these appraoches deserves a seperate paper, however i disagree.

Dear AC, dear reviewers,

We would like to kindly express our concern over the fact that, as we are entering the last 24 hours of rebuttal, two reviewers out of four are still inactive. As we worked hard to address your concerns in the past two weeks, it would be unfortunate to miss the chance to receive helpful feedback on it and improve the manuscript. Therefore, with this message, we gently ask reviewers Acn7 and Vx4x to engage in the discussion so we can make the best out of this last day of rebuttal.

Thank you very much in advance.