Learning Object-Centric Representation via Reverse Hierarchy Guidance

Thank you very much for your review of our submission.

About the model comparison in the paper

We have included a comparison of the two models mentioned in Weakness in the updated version of the paper. It is important to note that all of our models on the synthetic dataset are trained from scratch. For a fair comparison, AST-Seg-B3-BT is added to the table. In addition, the strategy of SLP on CLEVRTex-OOD dataset is different from us: we train the model on the original CLEVRTex dataset and evaluate on CLEVRTex-OOD without finetuning, while SLP split a training set from the OOD dataset and trains their model on this training set. Due to different training conditions, we did not record their OOD performance. We list the updated comparison in the table below:

Experiment on real-world datasets

In our original paper, we mainly demonstrate the validity of our proposed method through experimental results on synthetic datasets. Experiments on real-world datasets are included in supplementary materials to prove that our method can be generalized to more complex scenarios. Thank you for your suggestions, we found that the generalization of the model in real-world scenarios is worth paying attention to, so we added the experimental data of the model in real-world scenarios in the updated paper. We list the ARI-FG comparison in the table below:

Difference between other Object-Centric methods

Our main inspiration is to promote Object-Centric performance by incorporating some human visual system mechanisms into the network design. With an additional top-down pathway, we help the model learn more distinguishable features, as well as providing two alternative inference forms: a fast vision with a single bottom-up run, or a slow mode, running the network multiple times to get more accurate results. The experimental results show that our approach enables the model to learn more difficult samples, such as small or occluded objects in all the datasets, as well as texture-rich objects in CLEVRTex.

Thank you very much for your review of our submission.

Model Performance on Real-world Datasets

In the supplemental material, the performance of DINOSAUR is directly taken from the report in [1]. However, DINOSAUR doesn't release public models. Therefore, we are not improving on their basis. Instead, we obtain a baseline model by simply changing the input and reconstruction objective of the network to DINO features, which achieve a bit lower performance than reported in [1]. This reduces the performance gains from our method.

We have recently taken a closer look at the training tricks of previous work [1], for example, they reduce the dimensions of pre-trained features to lower dimensions at the input side and recover to the original dimensions at the output side. We find that these modifications can improve the clustering performance of the network with less computation. We apply these tricks to our model and retrain the model on real-world datasets. In the table below we provide a comparison of model performance before and after retraining.

After more thorough training of the network, we can come to two conclusions: (i) our performance benefits from better network settings, reaching +5.80% and +0.64% ARI scores compared with DINOSAUR; (ii) Compared with our reproduced baseline model, both Train-RHG and Infer-RHG bring remarkable performance gains.

About Object-IOU metric

In the supplementary material, we present the comparison of our method and other models in terms of Object IoU in CLEVRTex (Table 6 in supplementary material), where our method still outperforms other SOTA methods compared in the paper. The computation of Object IOU doesn't consider background pixels because we focus on the ability of the model to distinguish objects.

About object size division

We divide objects by their size distribution. Specifically, we count the number of pixels that all objects occupy in the ground truth, and we classify objects into small, medium and large on a ratio of about 1 to 5 to 1.

A scatter plot of object size vs IoU is a better way to express how the model's ability to discover objects varies with the size of objects. According to your suggestion, we have redrawn a plot and put it in the latest update, you can check it in the updated paper.

About hyperparameter choosing

Most of the hyperparameter settings follow previous work [3]. The main hyperparameter discussed in this paper is the weight of top-down conflict loss. The weight is 0 at the beginning of the training and gradually increases to a maximum value. This setup follows the intuition that the model needs to first learn to distinguish objects, and then enhance low-level features through top-down guidance to gain a stronger ability to discover objects.

The maximum weight of top-down conflict loss depends on the complexity of datasets. For simple scenarios, such as CLEVR, smaller weights are appropriate because increasing the weights will lead to a decrease in the accuracy of model reconstruction and segmentation. For more complex datasets, such as the CLEVRTex dataset, it is more necessary to find missing objects than to get more accurate segmentation, thus requiring a larger top-down conflict loss weight.

Mathematical description and typo error

We'll add a more detailed description of the symbols in section 3. The mapping $\mathcal{K}$ and $\mathcal{Q}$ doesn't change the dimensionality of features, still retaining $C=64$ as described in section 4.1. In addition, the summation index $K$ in Equation 4 appears due to a typo error, and it should be removed. In the updated paper, we have corrected these problems.

Conventional clustering approach

Considering the similarity of slot attention and k-means algorithm, we use k-means to cluster the low-level features output by the CNN encoder. We take the CNN encoder from two networks, namely RHGNet and BUNet in the paper, and cluster them respectively. After clustering the features of RHGNet, we obtain an ARI-FG score of 52.29%, while the clustering score of BUNet is 41.42%. The results verify that RHGNet learns better features.

[1] Bridging the gap to real-world object-centric learning

[2] Self-supervised Object-Centric Learning for Videos

[3] Object-Centric Learning with Slot Attention

We appreciate your review of our submission.

Pretraining confirmation

All the models on synthetic datasets are trained from scratch without using pretraining weight. Models on real-world datasets use the output feature of DINO to provide reconstruction objectives, and the pre-trained models have been indicated.

Weakness 1 & Question 3: Further explanation about Infer-RHG

The network can either quickly extract objects in the scene through a pure bottom-up inference, or it can run the network multiple times and search for a more precise perception through a top-down pathway. When N is set to 1, the network only runs the bottom-up process, which does not require more computational complexity, but still outperforms other SOTA models. When the top-down path is started during inference (Infer-RHG in the paper), the network requires more computation, and the performance will further increase. In the paper, every time we compare performance with other models, we list two performances that correspond to situations where N is set to 1 and 10 respectively.

The performance improvement of Infer-RHG comes from the selection of initial slot values. Random initial values in slot-attention will produce different attention distributions, eventually leading to performance differences, as shown in Figure 3 in the paper. This is similar to human visual perception, where sometimes we don't pay attention to objects with low saliency. Our model obtains multiple attention distributions through different initial slots so that these objects have a greater chance of being found. Using Top-down conflict as a signal, we pick out the perception that is most likely to find these objects, thus discovering small objects.

Weakness 2: RHGNet focuses more on object discovery ability

The top-down pathway demands the network to retain more general features because slots need to match all the features in the area they occupy, which causes slots to tend to reconstruct an "average" object that represents the overall features of the area while discarding more detailed features such as textures. This sometimes causes the model to converge to less precise reconstructions, but this doesn't always appear. We examined three models on ObjectsRoom trained with different random seeds. The model used for display had the problem described in Weakness, while the other two performed correctly, and all three models had an ARI score of 87% to 88%. We have updated the paper and replaced these figures with a correctly reconstructed version.

In addition, note that our displayed images are not carefully selected, but are directly chosen by taking out the first several images from test sets, thus the displayed result represents the average performance of our model. Among the 18 images shown in the supplementary material, a total of 107 objects appear, and our model misses 4 of them, which has made improvement compared with the current models shown in Figure 4 in the paper.

Weakness 3 & question 1: Working mechanism of our method

Conflict loss just drives $A$ to get closer to $M$. We add Information Integrity Loss $L_\mathrm{info}$ to avoid the situation that the network learns to decrease conflict loss by removing an object from low-level features. We use a shallow decoder to reconstruct the image from output features of the CNN encoder, and $L_\mathrm{info}$ is the reconstruction loss. This loss guarantees the existence of all objects in low-level features.

$A$ is initially random and is meaningless at this time, thus we set the weight of conflict loss to 0 at the beginning. Using only reconstruction loss, slot attention can discover objects in a simple scene in the form of object mask $M$. This preliminary perception may miss objects, but it is at least "broadly correct". We think of $M$ as a pseudo label and use it to supervise the CNN encoder: it learns a segmentation task through conflict loss. Despite given annotations with some errors, the encoder can still effectively learn object concepts, which in turn makes slot attention clustering easier, thus obtaining better object discovery results.

The authors of [1] experimented with the number of slot attention iterations during training, finding that more iterations may decrease performance. We list experimental data on the number of slot attention iterations from their paper in the table below:

This suggests that the depth of the network is not necessarily beneficial for learning Object-Centric representations. Our top-down path does not introduce additional depth to the network, but rather introduces a shortcut that directly connects the perception of the upper level (slot) with the lower level (CNN features), thus benefiting the learning of low-level features.

[1] Object-Centric Learning with Slot Attention

Thank you very much for your review of our submission.

We do not directly simulate the internal structure of human visual system, but rather achieve its function. In our main reference theory, Reverse Hierarchy Theory, the authors describe two main functions of human visual system, namely "Vision at a glance" and "Vision with scrutiny". The bottom-up and top-down pathways are responsible for the two kinds of vision. In our network, the top-down path will generate a conflict signal allowing the network to evaluate its output. The network selects a perception that minimizes the signal from multiple perceptions, thus obtaining as correct a perception as possible. At the same time, this process is optional, the model can not only obtain information quickly through pure bottom-up inference, but can also apply the top-down pathway for more careful observation and correcting perceptual errors. So, functionally, our networks are closer to the human visual system.