An Image is Worth $K$ Slots: Data-efficient Scaling of Self-supervised Visual Pre-training

1. The novelty of this work requires further justification. The approach is built iBOT and combines several existing techniques, such as iBOT and such as slot attention and cross-view contrast, without presenting substantial new contributions.
2. The categorization of datasets into object-centric, scene-centric, and non-object-centric may be oversimplified. For instance, the scene-centric dataset COCO can contain simple scenes with clear foregrounds.
3. It's difficult to determine whether the difference in model performance on different datasets comes from being object-centric or not, or other data distribution influences, including but not limited to object class distribution, image quality distribution, image resolution, etc.
4. The validations are still conducted on the previous datasets, making it difficult to determine how the training data affects the test data with different number distributions.

The paper's novelty is limited.

• Many insights in the paper have been discussed in previous work. For example:
  • "Features learned from NOC data can be linearly separable on ImageNet": Many works such as SEER[1], DINOv2 [2] (Table 2, uncurated data) have shown that pre-training on web data leads to features that perform well on ImageNet classification.
  • "NOC data is significantly beneficial for similar-domain downstream tasks": DINOv2 (Table 2) shows very good ADE20K performance with uncurated web dataset, Tolan et al. [3] shows DINOv2 trained on satellite images are better than those trained on ImageNet for an application on satellite images.
  • "non-object-centric data is rich in information with vast potential": a list of works showing good results with NOC data including SEER[1], AIM[4], SlotCoN, Vo et al. [5], ...
• The proposed SlotMIM is mostly a combination of iBOT (masked patch prediction) and SlotCon (cross view consistency + slot constrast)

Insufficient and incomplete empirical results

• Good SSL features have to work well on a wide range of downstream tasks and data domain. The paper considers only two benchmarks (ImageNet classification and ADE20K segmentation) to draw conclusions in Secs 3.2-3.4. Some fine-grained, retrieval or ood benchmarks should be included to have a more complete evaluations of the pre-trained features.
• Missing data points in Figure 5, 6 and 8: performance of features trained with DINO, DINOv2 and MAE on largest-scale datasets should be included to have the full pictures.
• The paper lacks a comparison to SlotCon. The performance of SlotMIM without masking and within patch loss should be added to Table 4.

[1] Self-supervised Pretraining of Visual Features in the Wild. Goyal et al., 2021.

[2] DINOv2: Learning Robust Visual Features without Supervision. Oquab et al., 2024

[3] Very high resolution canopy height maps from RGB imagery using self-supervised vision transformer and convolutional decoder trained on aerial lidar. Tolan et al., 2024.

[4] Scalable Pre-training of Large Autoregressive Image Models. El-Nouby et al., 2024

[5] Automatic Data Curation for Self-Supervised Learning: A Clustering-Based Approach. Vo et al., 2024

Object-centric terminology: Throughout the paper, the term "object-centric" is used to characterize datasets like ImageNet, whose images generally include a single object, and to contrast them to OpenImages, SA-1B, etc. that contain multiple objects. This terminology is not standard and may be confusing to readers, for example on L57 "our study reveals that several insights from object-centric learning remain applicable to NOC data", where "insights from object-centric learning" are derived from DINO, IBOT, SimCLR, etc. trained on ImageNet. Also, many related works describe datasets like COCO as "object-centric" and claim that ImageNet is not "object-centric" enough to train object-centric models on, which is the opposite of the terminology used in this paper. To avoid the confusion, the authors could consider using terms like "single-object" or "multi-object" datasets instead.

Other terminology: Other aspects of the chosen terminology are also non-standard and confusing. A few examples:

• L120 "normalizes attention scores on the query side instead of the key side" (axis?)
• L152 "assigns each patch token a soft one-hot encoding" (can't be both soft and one-hot)

Vague claims: The paper makes several claims that are not supported by the experiments or existing literature. For example:

• L182-185 "A key factor contributing to the lack of semantic meaning is that the iBOT loss LiBOT is computed between patches within the same view. Consequently, there is no explicit guidance for learning invariant representations across different views of the same object or scene." However, self-supervised works like I-JEPA (https://arxiv.org/pdf/2301.08243) demonstrate that predicting missing patches in latent space leads to learning semantically meaningful representations.
• L224-226 "This worked well for ResNet since the network architecture provided strong inductive bias for objectness." What are the inductive biases towards objects in a convolutional network? On the contrary, previous papers like Slot Attention (https://arxiv.org/abs/2006.15055) build on the idea that CNNs lack such biases and proposed to add them explicitly.
• L533-535 "To harness this potential, we formalized learning from NOC data into two sub-tasks: scene decomposition and object-centric representation learning." The experiments in the paper do not demonstrate the effectiveness of this formalization in sub-tasks, nor that the proposed architecture and loss implement the sub tasks separately.

Limited novelty and missing comparison with previous work: The loss formulation with cross-view patch alignment is similar to that of VicRegL (https://proceedings.neurips.cc/paper_files/paper/2022/file/39cee562b91611c16ac0b100f0bc1ea1-Paper-Conference.pdf), which is not cited or compared against in the paper. Furthermore, most components are simply imported from existing work:

• The patch-wise loss is that of IBOT
• The strategy of pooling patch features based on prototype assignments is from SlotCon
• The contrastive loss for the slots is again from SlotCon and MoCo Undoubtedly, a novel combination of such components and a thorough analysis of their effect would be valuable, but the paper lacks in this aspect.

Insufficient evaluation: L310-312 "under ImageNet fine-tuning setting, the top-3 methods (BEiT, SplitMask, and SlotMIM) have best performance when trained on COCO+ instead of ImageNet." The gap in performance is around 0.1% which is within margin of random fluctuations, no conclusions can be drawn from this. If the authors want to provide a convincing argument, all experiments should be repeated multiple times and reported with confidence intervals.

Evaluation protocol: A few choices in the evaluation protocol are questionable:

• L264-265: "For ImageNet fine-tuning, all models use the average-pooled token." This is not standard practice: if a model is trained with a CLS token the expectation is that the best global representation for a classification task is in the CLS token, not the average-pooled token.
• All pre-training datasets are limited to 241k images, which is a small fraction of what those models are usually trained on, and for which their training recipes are optimized. From the ImageNet linear probe results in figure 3, all the models seem undertrained, which makes all comparisons invalid.

• Novelty: The novelty of the approach seems limited, as it primarily combines slot attention and iBOT.

• Performance: The paper lacks tables comparing the method with state-of-the-art (SOTA) visual representation learning methods. From the plots in Fig. 4, the performance appears poor, with approximately 84% ImageNet top-1 accuracy and around 49% ADE20K segmentation mIoU, although the model may not have been sufficiently trained. Additionally, SOTA representation learning methods, such as EVA, EVA-02, and InternImage, are not compared in the paper.

• Motivation: The paper claims to focus on data-efficient representation learning, yet it uses 241,000 images for training, which is still a large number. Visual representation learning is a data- and computation-intensive domain, and it tends to be winner-take-all, with users only downloading and using the most effective models while ignoring insufficiently trained ones.

• Experiments: If the paper aims to emphasize its object clustering performance, it should report unsupervised object segmentation results. Clarity: The paper fails to clearly illustrate its differences with DINOv2 and iBOT. A figure comparing the overall framework with related works could be added.

• Writing: The phrase "enhance MIM with cross-view consistency regularization" is unclear; what does "cross-view" mean in this context?