Competition Priors for Object-Centric Learning

Novelty

• All components of COP are not new. The non-iterative attention operation is common. For example in SAVi, they also perform single iteration Slot Attention. The primitive slots come from image features, which are conceptually similar to the conditional initialization in SAVi, with the only difference that they are not from initial frame hints
• Is iterative attention a feature or a bug? To me, iterative Slot Attention enables OCL models to decompose complex data as seen in follow-up works such as STEVE [1] and LSD [2]. It is unclear to me if the non-iterative SWTA Attention can handle more complicated images (see Experiments below)

[1] Singh, Gautam, Yi-Fu Wu, and Sungjin Ahn. "Simple unsupervised object-centric learning for complex and naturalistic videos." NeurIPS, 2022.

[2] Jiang, Jindong, et al. "Object-centric slot diffusion." arXiv preprint arXiv:2303.10834 (2023).

Experiments

• The datasets used in this paper are too simple. OCL has witnessed tremendous progress in recent years, where OCL methods have proven effective on more complex datasets [3]. The simple images tested in this paper make it hard to assess the capacity of COP. A necessary experiment is to incorporate COP with better OCL models such as STEVE, and test it on complex datasets like MOVi
• The experiments in the paper mainly focus on object segmentation. While it is an important outcome of OCL, the quality of learned object slots is another important aspect. I would suggest the authors to at least perform an object property prediction experiment on CLEVR following the protocol of Slot Attention

[3] Seitzer, Maximilian, et al. "Bridging the gap to real-world object-centric learning." ICLR, 2022.

1. The baselines used in the experiments are rather weak. The authors mention Implicit Slot Attention [1] in section 3.3 when discussing scalability, but do not compare against it in the experiments.
2. Along the same lines, it would have been informative to compare with other methods that focus on slot initialization such as a version of Slot Attention using learned slot initializations or BO-QSA [2].
3. Section 3.3 uses Implicit Slot Attention [1] as an example of a model using up to 11 iterations, but this is misleading since the experiment on number of iterations from that paper was to show that Implicit Slot Attention is robust to increasing number of iterations when compared with Vanilla Slot Attention. While it can scale to more iterations, their method does not require more iterations to perform well.
4. Since vanilla Slot Attention already does quite well on the datasets used in the paper, it would be interesting to see how COP performs on a more complex dataset such as CLEVRTex.
5. It is not clear to me from the experiments the importance of max-pooling or if some other method to learn primitive slots from the input would also work well. This is a central part of the algorithm and it would strengthen the evidence for using max-pooling if this is compared with other ways of pooling such as average-pooling or strided convolutions.
6. (minor) The original Slot Attention does not set the temperature to $\sqrt{n}$, but instead it uses the sqrt of the slot dimension.

[1] Object Representations as Fixed Points: Training Iterative Refinement Algorithms with Implicit Differentiation. https://arxiv.org/abs/2207.00787

[2] Improving Object-centric Learning with Query Optimization. https://arxiv.org/abs/2210.08990

Section 3.1 is badly structured: if the encoder and the decoder are exactly the same as Slot Attention please just refer to the original paper to save space and avoid confusion. Then, use the extra space to explain better the novelty in the object-centric bottleneck (see paragraph below).

The mechanism of Soft-Winner-Takes-All (SWTA) is not explained clearly and the comparison with Slot Attention is misleading. Slot attention takes the softmax over the queries first and then normalizes over the keys by dividing each row by its sum. As I interpret the paper, SWTA seem to perform the first operation, i.e. the softmax over the queries, but skips the normalization. Therefore, what is the novelty? Skipping the normalization? Running a single iteration instead of multiple ones? In both Slot Attention and SWTA there is competition for the patches between queries: if a patch is assigned to one query/slot it can not be assigned to others.

I strongly disagree with the analysis performed in section 3.2 about the "competition through MaxPool layers". In particular, the discussion about "sub-networks" in the following sentences: "This results in sub-networks competing to have higher activations. During back-propagation, units that win and the subnetworks that are responsible for this will get updated. As a result, a winning sub-network is reinforced to win more if it predicts correctly." I'd like to point out the CNN layers before MaxPool share the same weights across locations therefore there are no sub-networks in competition, only one network. I would like the authors to provide references to the "sub-networks" interpretation to support their claim in the paper.

In the same paragraph, it is said that "A neuron has a higher chance of winning if it explains a different part of the input, rather than explaining the same feature as another neuron." I argue that the final "primitive slots" are diverse simply because they are pooled from different regions of the input image by means of local CNN filters and local max pooling as shown in Figure 2. This is a well-known property of CNNs and it also introduces an implicit bias on the shape and size of the objects which the authors do not discuss.

Other than the query generation process the discussion about competition in section 3.1 does not add much to the original Slot Attention paper: SWTA seems to be a slight variation of their competitive attention implementation and the decoder is identical. If other papers already discussed the importance of competition in those two mechanisms, why repeating the same arguments here?

The number of slots is hardcoded and manually tuned for each dataset. This is a limitation of this method that doesn't make it stand out compared to previous ones. If anything, creating the "primitive slots" by means of max pooling is even more limiting than the sampling mechanism of Slot Attention.

Experiment design:

• The experiments only include very basic datasets where even basic color-based clustering methods would suffice. Since a few years, the object-centric community has moved on to more challenging datasets such as ClevrTex and MultiShapeNet, where the proposed method would be more interesting to evaluate.
• Also, I disagree with the choice of evaluating only the FG-ARI metric, while segmentation is a good proxy for object-centric learning, it does not capture the full complexity of the task. Other works try to evaluate the quality of the slots by means of downstream tasks such as object tracking or property prediction.
• Finally, the ablation study is not very informative. The authors should have compared with vanilla Slot Attention where 1) the query generation step is replaced with their max-pooling-based method, and 2) the iterative attention is replaced with a single attention layer, while keeping the rest unaltered. This would have provided a fair comparison and a better understanding of the contribution.

The abstract claims "scalability" as a feature of the proposed method, which I can not agree with. See the related question below.

Public code: the code in the attached zip archive is broken. For example, train.py tries to call a non-existing function read_cli_args_and_get_config. This way, it's not even possible to check that the code reflects the method described in the paper because there are multiple implementations in the zip archive. I appreciate the effort of including the code in the submission, but it should be double-checked beforehand and possibly come with some instructions.

Overall manuscript quality: many sentences are poorly written and the text would require a thorough revision. A few examples:

• "Similar to the encoder, we build the CNN layers in such a way that they preserve the spatial dimension. But, the MaxPool layers reduce it."
• "Primitive slots" are capitalized in the figure captions but never in the text. Sometimes there is a dash between the two words, sometimes not. Also it's unclear whether "primitive slots" are a new concept and how they differ from the usual term "queries", the first mention in section 3.1 could be more explicit.

1. Scope of Dataset Evaluation: The empirical validation of the COP model is confined to synthetic datasets. While the authors posit that COP's attributes render it highly scalable for larger datasets, the absence of real-world dataset assessments renders the claims of its object-centric representation less compelling.
2. Contemporaneity of Baselines: The baselines utilized in the study are somewhat dated. A comparative analysis with more recent methodologies would be beneficial for a comprehensive evaluation, and such comparisons should be reflected in the results table.