DisFormer: Disentangled Object Representations for Learning Visual Dynamics Via Transformers

Weakness:

1. We used OBJ3D which has similar visual complexity of CLEVRER. We are still working to compile our numbers on CLEVRER and hope to shape them during discussion phase.

2. We are working on specific ablations and hope to shape them during discussion phase.

3. We have provided disentanglement evaluation in appendix.

4. Added in the appendix.

1. We would like to point out to reviewer that Sysbinder discovers object representation for images and we would like to discover object representations for video. We still performed used frame by frame sysbinder and found satisfactory results on 2D BS dataset.
2. We are working on compiling example images and hope to post them in the discussion phase.
3. We are working on high level diagram and hope to post them in the discussion phase.

1. That's very interesting suggestion. We are currently working on it and hope to add results in discussion phase.
2. We found that comparing weights of concept vectors result in faster convergence. The $\hat{w}_{t}^{i,b,j}$ is the predicted weight of $b^{th}$ block and its jth concept vector. It is output of transition model.

1. We are working on high level diagram and hope to post them in the discussion phase.

2. OCVT’s alignment algorithm is based on hungarian matching algorithm over L2 norm of latents. In our experiment it performed poorly. We hope to add some results during discussion phase.

3. We have added the details in the appendix section

4. Each block has it’s own set of concept vectors. For instance the block(s) representing color would have concept vectors as {red, green, blue} etc. whereas those representing size would have a different set of concept vectors. two different attributes cannot be a linear combination of same concept vectors. Thus concept C is not shared across all the blocks.

5. k is number of concept vectors

6. Yes it is output corresponding to $s_{t}^{i,b}$

7. Yes the reviewers understanding is correct.

8. We have added position error computation section in appendix

9. In our choice of dataset the color and shape are correlated and are not represented by separate blocks.

1. Internal representation of mask does not have appearance information of object. We thank the reviewer for the suggestion. We would also like to point out that we also need the appearance information of the object and only the mask's hidden representation will not be sufficient. We are working on the reviewers suggestion and hope to share results in the discussion phase.

2. The line 10 and 11 in Algorithm 1 does the projection of blocks to concepts. As the concept vectors for a given block common across the objects these add strong prior for learning disentangled representation.

3. We performed a grid search of hyperparameters for a reasonable range. We found that DisFormer’s training is stable in the range of consideration.

4. We started with hyperparameters reported by the baselines and tuned in limited space.

5. We added a section in appendix describing this.

6. In section 4.3 single model was trained on two datasets. The first dataset have all small objects and second dataset have all big objects in video. During testing we test on a dataset where some objects are small and some are big.

1. Most of our model components are dependent on each other, so the entire system may not work if we remove any of those. At the same time, if the reviewer has specific suggestions on which ablations to try with respect to moving out certain components, we will be happy to report those. There was a typo in experiment section 4.2 where instead of object extractor we replaced block extractor with MLP.

2. This is a very important suggestion. We are working on compiling a list of failure cases of our model. We hope to upload this to these before the end of the discussion phase.

3. In our ablation experiment, we rigidly assigned concept weights for blocks to values of 0 and 1. Unfortunately, the decoding process with these blocks failed to produce any meaningful images. This experiment has conclusively shown that, under these conditions, individual concept vectors do not effectively represent interpretable attributes. About the second part of the question we are working defining these sepration. We hope to share the results before the end of the discussion phase.

4. We have added the appendix section of disentanglement evaluation.

We have addressed the need for a permutation model by including a specific example in the newly added content within the appendix section C. Furthermore, we have enhanced and clarified the details in section 4.4 based on your feedback.

1. We are unsure of the specific claim that the reviewer is referring to. To the best of our knowledge, we have not made any claim regarding our ability to do well with deformable objects. If the reviewer can point out a specific instance, we would be happy to correct it. While we agree that we have experimented only with rigid body surfaces, we would like to to argue that disentanglement is a hard problem, and some of the SOTA techniques (Symbinder refer) have only worked with images, which we have extended to video in our work. Further, SlotFormer [] (which is a very recent work) for video prediction, also works on only simple rigid-body dynamics. Hence, we strongly believe our setting is commensurate with some of the recent works in this area.

2. In response to the reviewer's suggestion, we have incorporated additional evaluation and experiments illustrating the decoupling of blocks in the appendix section F. Importantly, we emphasize that these blocks are learned in a fully unsupervised manner. One such instance is the block that has learned to represent color, as demonstrated in section 4.4. However, the final part of the review is currently unclear to us, and we would appreciate further clarification.

3. That was a typo and it should have been "by replacing the block extractor”. We have fixed it in the paper.

1. Thanks for pointing this out. We have added a task description in the introduction. While we agree that we have experimented only with visually simple datasets, we would like to to argue that disentanglement is a hard problem, and some of the SOTA techniques (Symbinder [1]) have only worked with images, which we have extended to video in our work. Further, SlotFormer [2] (which is a very recent work) for video prediction, also works on only simple rigid-body dynamics. We are working on compiling example images and hope to post them in the discussion phase.

2. We thank the reviewer for the suggestion and we agree with reviewers rephrasing. We would like to point out that we are not taking attention to the entire image and rather take attention to the image masked out by an object mask. We are aiming for disentangled representation of an object and thus we add that prior by inferring blocks which is iteratively refined by taking attention over latent object representation.

3. We want to highlight to the reviewer that the permutation gadget undergoes training in a self-supervised manner and we don’t have ground permutations. Its sole purpose is to align predicted object blocks with ground object blocks for dynamic loss computation. Importantly, during testing, there is no necessity for loss computation, and therefore, the permutation module is excluded from the testing process. We have added more details in the appendix.

4. We have included the specifics of object extractor training in the appendix. Additionally, it's worth noting that for the 2D environment, we employ Mask R-CNN as the object extractor. The training of Mask R-CNN necessitates supervised data, which is generated in unsupervised fashion from expert models.