Semantic-Guided Consistency and Discrimination for Siamese Representation Learning

a) Novelty is a major issue of this work, the only technical contribution is to introduce foreground/background separation into self-supervised learning, which however, has been studied in multiple earlier works and is applicable to a limited scope.

• Separating the foreground using saliency or other segmentation/localization techniques for representation learning has been validated by a series of early works, simply listing some of them: CAST[a], InsLoc[b], MaskContrast[c], ContraCAM[d], VADeR[e], Hier. Group[f].
• More advanced works further consider the separation between objects (object discovery), eg, ORL[g], SoCo[h], SlotCon[i], Odin[j], CrOC[k], and COMUS[l].
• Or show superior object detection performance using only pixel-level supervision: PointCL[m].
• Yet this work is still limited to foreground/background discrimination, which seems good for object-centric images like ImageNet, but may find it hard to generalize to scene-centric or even web-crawled uncurated images [n,o].

b) I also noticed several incomplete comparisons, eg, tab.5 cited DenseSiam & SlotCon, yet simply overlooked them for comparison. Besides, across detection-related benchmarks, a complete comparison with ORL, SoCo, SlotCon, Odin, and PointCL is needed.

c) In modern self-supervised learning architectures (eg, DINO[p], DINOv2[q], GroupViT[r], and object discovery-related literature as above), the emergence of objectness, or their separation from backgrounds has been a common property when suitable backbone (eg, ViT) or/and supervision (representation bottleneck) is applied. Thus there is concern that the separation of foreground/background is trivial for modern learning frameworks, and questions the necessity of an explicit separation process.

References:

[a] CASTing Your Model: Learning to Localize Improves Self-Supervised Representations, CVPR 2021.
[b] Instance Localization for Self-supervised Detection Pretraining, CVPR 2021.
[c] Unsupervised Semantic Segmentation by Contrasting Object Mask Proposals, ICCV 2021.
[d] Object-aware Contrastive Learning for Debiased Scene Representation, NeurIPS 2021.
[e] Unsupervised Learning of Dense Visual Representations, NeurIPS 2020.
[f] Self-Supervised Visual Representation Learning from Hierarchical Grouping, NeurIPS 2020.
[g] Unsupervised Object-Level Representation Learning from Scene Images, NeurIPS 2021.
[h] Aligning Pretraining for Detection via Object-Level Contrastive Learning, NeurIPS 2021.
[i] Self-Supervised Visual Representation Learning with Semantic Grouping, NeurIPS 2022.
[j] Object discovery and representation networks, ECCV 2022.
[k] CrOC: Cross-View Online Clustering for Dense Visual Representation Learning, CVPR 2023.
[l] Unsupervised Semantic Segmentation with Self-supervised Object-centric Representations, ICLR 2023.
[m] Point-Level Region Contrast for Object Detection Pre-Training, CVPR 2022 (best paper finalist).
[n] Demystifying Contrastive Self-Supervised Learning: Invariances, Augmentations and Dataset Biases, NeurIPS 2020.
[o] Divide and Contrast: Self-supervised Learning from Uncurated Data, ICCV 2021.
[p] Emerging Properties in Self-Supervised Vision Transformers, ICCV 2021.
[q] DINOv2: Learning Robust Visual Features without Supervision, arXiv preprint.
[r] GroupViT: Semantic Segmentation Emerges from Text Supervision, CVPR 2022.

One critical aspect that requires further clarification is the differentiation between this work and the closely related LEWEL. Both papers share the common objective of mitigating the influence of object-irrelevant background information in representation learning. However, the exact distinction between the two approaches needs to be more clearly delineated. Specifically, how does this proposed method differ from LEWEL in terms of feature-level salience maps and their utilization in the learning process?

While the experiments demonstrate that the proposed method outperforms LEWEL, it's important to acknowledge that the margin of improvement appears relatively small. For instance, the difference in accuracy between LEWEL-MoCo and SCD-MoCo is only 0.2 (asymmetric loss) and 0.3 (symmetric loss). The performance gain in PASCAL VOC and COCO is also less than 0.4 with asymmetric loss. It would be helpful if the authors could provide insights into why this marginal difference is significant.

1.The novelty of this work is limited. 2.The performance gain of this work is weak. 3. In Table 1, there is only linear probing performances. 4. Several works are relative closely to this paper, such as *On the Importance of Asymmetry for Siamese Representation Learning *Unsupervised Semantic Segmentation by Contrasting Object Mask Proposals *Unsupervised Learning of Dense Visual Representations *Distilling Localization for Self-supervised Representation Learning *Reverse attention for salient object detection please detail the differences between the proposed method and them one by one.

• The topic seems to be out of date and the novelty is limited. How to build a stronger MoCo or BYOL is hotly studied about 2-3 years ago and there has been a lot of works attempting to solve this problem from various perspectives. Distinguishing fore- and background to make the training more stable and accurate is not a new idea. As far as I know, there have been many methods doing similar things, including but not limited to [1, 2], though they may not be the same. Besides, recent mask image modeling (MIM) methods, represented by MAE [4], have pushed the performance of self-supervised learning to a new higher level. What is the advantage of the proposed method over MIM ones?
• Some more recent and important related works are missing. As listed above, the missing related works need to be discussed and compared.
• Even with the missing works not listed and compared, the performance advantage over the listed older methods is still too marginal.
• The training cost is not analyzed. Though some performance promotion is obtained, how much the training cost is increased needs to be clarified.
• Why the proposed method can work should be analyzed and demonstrated in the main text. At least, the visualization analysis in the appendix is more necessary to be moved into the main text.

[1] Mishra S K, Shah A, Bansal A, et al. Object-aware Cropping for Self-Supervised Learning[J]. Transactions on Machine Learning Research, 2022.
[2] Wang X, Zhang R, Shen C, et al. Dense contrastive learning for self-supervised visual pre-training[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021: 3024-3033.
[3] Yang C, Wu Z, Zhou B, et al. Instance localization for self-supervised detection pretraining[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021: 3987-3996.
[4] He K, Chen X, Xie S, et al. Masked autoencoders are scalable vision learners[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 16000-16009.

1. The main point of the paper is to extract pure foreground view in random cropping, however, the mechanism of why a simple MLP (the saliency map network) can help is not well-clarified.
2. The effectiveness of SCD also relies on newly extracted foreground and background feature maps, which is a special case of new views (cropped at the same location with different augments). Whether using additional crops at the same location with different augments can also boost self-supervised learning? Or in other words, multi-crops is more important instead of the separation of foreground/background?