Looping LOCI: Developing Object Permanence from Videos

Thank you for acknowledging that Loci-Looped exhibits strong object tracking performance. Please see the general response as response to most of your comments.

Indeed, in Figure 4 the model narrows down the hidden object as it approaches the edge of the occluder. This interesting behavior is fully emergent as the model never encountered vanishing objects in the training dataset. The model does not completely let the object vanish, which becomes apparent by the surprise signal.

In general, this inner loop should be applicable to other compositional scene representation learning methods if they use a latent prediction module. For example this is the case in SAVi, where a percept gate could be incorporated after the Correction module. In contrast, this would not work for Steve as this model does not predict the next object states. However, we hypothesize that the success of the inner loop in Loci-Looped also depends on its inductive bias to maintain stable object representations which is enforced by the GateLORD architecture as well as by the regularization terms.

Thank you very much for the thoughtful comments.

Thank you for the helpful comment, pointing out that the slot decomposition is missing in the illustrations. We now provide illustrations of the slot decomposition in the appendix. We hope this makes clear, that the objects persist in the slot representation when occluded. Moreover, we now provide videos in the supplementary material.

Concerning the importance of the different regularization terms: The Gestalt Change, the Position Change and Object Permanence Regularization are ablated in Traub et al. (2023), suggesting that the Gestalt Change and the Position Change regularization is important for learning. We are currently running ablation studies on the Input-Frame Reconstruction Loss and the Gate Opening Regularization and will add them to the Appendix.

We now clarify in the methods section that we define intertia as the continuation of motion unless acted on by an external force. We test this property by letting the model predict how the occluded objects continue to move when hidden or when a blackout occurs.

Indeed non-rigid materials present a challenge to the current model (and very likely to most others). We are happy to include this aspect as a limitation in the discussion section among other limitations.

Thank you very much for acknowledging that the addition of the inner loop substantially improves tracking for occluded objects.

Thank you for the comment. Indeed, the tested occlusion scenarios are rather simple. The more surprising it is that these tests can not be solved by current state-of-the-art models like Loci-v1 or SAVi. Although our introduction of the internal information fusion process is incremental, our work clearly shows that this addition has a major effect on the model's ability to deal with occlusion scenarios. To the best of our knowledge this is the first time such a process is introduced for compositional scene representation models.

In general, the inner loop can be incorporated in all models that make use of a latent prediction model. For compositional scene representation models, this is however not very common as many models do not ensure slot consistency over time using recurrent next-step predictions. Incorporating the precept module into other compositional scene representation models is an interesting avenue for future experiments. The slot-wise information fusion process may also be an interesting candidate for other object-centric transition models in different model domains, for example in the DREAMer (Hafner et. al, 2019) architecture.

We deeply value the reviewer's recognition of our model's ability to emulate infants' holistic learning without supervision -- a pivotal aspect in understanding how these innate abilities can be jointly learned.

We appreciate the suggestion for a rule-based baseline. We have added a Loci-Visibility baseline, whose percept gate is controlled solely based on the perceived occlusion state i.e. Loci switches to the inner-loop when objects become occluded. We find that it performs superior to Loci-Unlooped highlighting the importance of the inner loop. However, it performs inferior to Loci-Looped, demonstrating the importance of learning an adaptive gate control function that flexbily balances the inner and the outer looop, rather than approximating a simple rule.

We also recognize the necessity of testing the model on more complex occlusion scenarios, such as non-linear object movements or containment situations. This limitation has been now duly addressed in our discussion, emphasizing the need for datasets that pose greater challenges in terms of segmentation, localization, and tracking to further evaluate and enhance the model's abilities.

Thank you for pointing out that Figure 3 was a bit confusing in that respect. Due to averaging effects the greater surprise in the vanishing trials was not explicitly illustrated in the plot. We have added an additional box plot (Figure 3a) showing that the maximum slot errror is significantly larger when hidden objects fail to reappear after occlusion and after the occluder rotates to the ground, compared to trials in which objects do reappear. We also now add a video to the supplementary material, which shows that our system expects the reappearance and then "parks" the object behing the occluder until the occluder falls over. Please note that this latter behavior is fully emergent, as our system was not trained on vanishing trials.

Answering the questions: The requirement that the object mask must be contained within the object mask itself stems from equation 1. This constraint emerges from our approach to computing the object mask, where we consider only slot-object k within the scene, disregarding other slots. Consequently, slot k only competes with the background for visibility yielding the object mask. Conversely, when deriving the visibility mask, slot k competes not just with the background but also with all other slots present. If the other slots do not intersect with slot-object k, the object mask aligns with the visibility mask. However, if there's an overlap between the remaining slots and slot-object k, the visibility mask becomes a subset of the object mask. This implies that the visibility mask can never exceed the object mask (paragrpah 3.2.1.).

Indeed, the sentence "By adding Gaussian noise with a fixed standard deviation ..., learning is biased to move further into plateaus away from ridges where possible", refers to the landscape of the rectified tanh activation function. We rephrased this sentence to "... the gates tend to either close or open, rather than remaining partially open", to improve comprehension.

The distinction between L0 and L1 regularization lies in the computation of the backpropagated error. In the scenario of the L1 Loss, the backpropagated error equates to the value of alpha (gate opening; alpha * 1). However, in our case, the backpropagated error is either 1 (when alpha > 0) or 0 (when alpha = 0, expressed as heavyside(alpha) * 1).

Thank you for pointing that the correlation needs more explanation. The concept here revolves around the similar trajectories of objects reappearing and vanishing. The visibility spike of reappearing objects (observed after frame 30 in Figure 3) signifies an expected moment when both vanishing and reappearing objects should become visible again. Therefore, the visibility of reappearing objects serves as an indicator for the visibility of vanishing objects. Our observation reveals that both the slot error and the visibility mask for vanished objects rise within this specific time interval. This suggests that the model anticipates the vanishing objects to become visible again during this interval, effectively predicting their reappearance. We've revised the paragraph to convey this argument more clearly and moved it to the appendix.

Indeed, the occluding plate is also assigned a slot. It is represented as one object in the model. We have added illustrations to the appendix showing the model's prediction for the falling plate. Moreover, we have added video material to the supplementary material.

First of all, we want to express our appreciation to the reviewers' and area chair's detailed, excitingly positive, and constructive feedback and the time they have spent to improve our work. We worked hard to increase the conciseness and comprehensibility of our manuscript. In the following, we respond to the general points that were addressed by multiple reviewers. Please find our answers to specific questions in the responses to the individual reviews.

A recurring concern raised by several reviewers centered on the scalability and generalizability of our model to more complex datasets. We acknowledge this concern as an important step for future research. However, our work is about developing object permanence from video in a fully unsupervised fashion and without any prior information about objects whatsoever. None of the current other state of the art systems (including those mentioned by the reviewers) are able to accompolish this. Besides, in Traub et al. (2023) it was shown that Loci is able to handle more complex objects with diverse texture patterns. In the short time it is unfortunately not possible for us to create a suitable dataset and probe this ability in this paper. However, we refer to a recent related publication on Loci, which demonstrates largely superior scene segmentation performance in the MOVi-e dataset.

Another shared concern raised was the model's inability to handle moving cameras, an aspect not explicitly addressed in our current architecture that primarily focuses on static camera poses. We acknowledge this as an intriguing extension that we've highlighted in our discussion section. We anticipate that the perceptual gating mechanism could seamlessly align with a model accommodating camera motion. Utilizing motion signals, derived from optical flow or provided as command signals, offers a promising approach to dynamically update predictions in line with camera movements. This adaptation aligns predictions with updated sensations, allowing the percept gate to function as intended in ensuring correspondence between predictions and sensory input. Moreover, note that a recent related paper reports that Loci yields largely superior performance on the MOVi-e benchmark which also includes non-stationary camera poses.

Lastly, missing baselines was criticized by the reviewers, for example lacking comparisons to SAVi++ and Slotformer, besides our comparison with SAVi (which shows that Loci-Looped clearly outperforms SAVi). In our understanding these comparisons are of limited use. SAVi++ main extension is its improved performance on real-world datasets, incorporating camera motion and explicitly exploiting ground-truth depth information in training. Neither of these characteristics apply to our study of object permanence and our datasets. Moreover there is no architectural improvement from SAVi to SAVi++ that would adress the problem of maintaining stable slot representations of temporairly hidden objects, suggesting that the performance of SAVi is a good indicator on how SAVi++ would perform on our tests. Slotformer, on the other hand, is not a compositional scene representation model but a slot-based video prediction model that trains and relies on pre-computed slot-representations, for example, computed using SAVi or Steve.

Concerning the intuitive physics models: We were not able to train PLATO on the ADEPT vanish scenario (as also stated in Piloto et al. (2022)), because the model expects aligned input masks that need to be provided consistently. In addition, PLATO requires a very coarse temporal resolution (15 frames for one video) simulating only short occlusions, whereas Loci-Looped and SAVi can be trained on fine temporal resolutions (41 frames) simulating longer occlusions. We did not include the ADEPT model as baseline as it would be a skewed comparison in our opinion. The model depends on supervised information to train its encoder, its decoder and its particle filter. Moreover, the ADEPT model uses an out-of-the-box physics engine. We did not include baselines without explicit object representations as numerous related work suggest that object agnostic models perform inferior (Piloto et al. (2022), Smith et al. (2019) , Wu et al. (2023), Villar-Corrales et al (2023)). We now give a detailed explaination on our choice of baseline models in the appendix.