Multiple Object Stitching for Unsupervised Representation Learning

Thanks for your constructive comments. We address the concerns as follows:

• More introduction of related work

  Thanks for your constructive comments. Following the suggestions, we supplement more related works in the Introduction section.

• How to use the proposed strategy to obtain object-related content?

  The object-related images can be easily obtained from the off-the-shelf classification datasets, such as CIFAR10, CIFAR100 and ImageNet-1K. Due to single category label of each image from these datasets, the image context of these datasets can be regarded as single-object one.

• Discussion about advantages and disadvantages

  We supplement an additional section in the revision to discuss the advantages and disadvantages of our proposed method. The clarifications are presented as follows.

  Our proposed method synthesizes multi-object image by stitching off-the-shelf single object centric images, inevitably introducing artificiality produced by the boundary between stitched objects. Therefore, there is potential domain gap between our synthesized multi-object images and the natural ones, which may affect the multi-object representation learning. To this end, we introduce the contrastive objective between natural single object centric images to alleviate the above issue, which effectively improves the multi-object representation performance in the experiments.

• The increased computational cost of our method

  Actually, stitching more images doesn't introduce additional computational cost. Specifically, all stitched images are resized into 224×224 size, the same image size as the one adopted by plain contrastive learning methods. Hence, compared to plain image view for contrastive learning, our stitched images doesn't increase the cost of computation.

• Training curves of our method

  We supplement the pretraining curves in the appendix of our revised paper (E Pretraining Curves section). The clarifications are presented as follows.

  To further understand our method, we present the pretraining curves of our total loss and all loss items in Figure 6 and Figure 7. For CIFAR100 dataset in Figure 6, all losses achieves stable convergence during pretraining. Notably, the value of single-to-single loss is larger than the ones of multiple-to-single and multiple-to-multiple losses. I analyze the above results as follows. Due to the supervision of multiple-to-single and multiple-to-multiple losses, the representations extracted by out method are encouraged to capture the similarities between different image instances, e.g., potential positive samples. Therefore, single-to-single loss cannot simply reduce the similarities of other potential positive pairs (not the given positive pairs) for lower loss value.

  For more complicate ImageNet-1K dataset in Figure 7, the values of losses don't consistently decrease with the increment of pretraining epochs. However, we find that the performance of its representation keeps sustained improvement during the pretraining. We plan to further explore the insight behind this result in future work.

• T-SNE visualization results

  We supplement the T-SNE visualization results in the appendix of our revised paper (F The Visualization of Pretrained Representations section). The clarifications are presented as follows.

  To further assess the representation quality of our method, we visualize the representations of our method and iBOT using t-SNE. For better visualization, we select the representations of simple CIFAR10, which only contains 10 categories. For more complicated ImageNet-1K and CIFAR100 datasets, it's difficult to discriminate massive categories for better visualization by limited distinguished colored markers. As shown in Figure 8, our method can well group the samples from the same categories in the representation space, in spite of without supervised training. Compared to iBOT, our method achieves better representation consistency for "green" category.

• The evaluation results on object detection and semantic segmentation

  Indeed, we have evaluated our pretrained model on object detection and instance segmentation tasks of COCO dataset in Table 3.

  To further validate the effectiveness of our method, we conduct transfer learning experiments with the pretrained ViT-S/16 backbone using FPN architecture on semantic segmentation of ADE20K dataset. All models are finetuned for 40k iterations. The results are listed as follows.

  Method	mIoU	aAcc	mAcc
  SelfPatch	41.2	80.7	52.1
  ADCLR	42.4	81.1	54.2
  MOS (ours)	43.9	82.4	55.8

  The experimental results demonstrate that our method significantly outperform the previous method.

Thanks for your constructive feedback. We handle your main concerns as follows:

• More introduction about related works

  Thanks for your constructive comments. Following the suggestions, we supplement more related works in the Introduction section and further refine the writing.

• The addressing of the issue about random cropping in multi-object images

  There are some misunderstandings of our proposed method.

  Our proposed method is designed to avoid random cropping issue suffered by directly conducting contrastive learning methods on natural multi-object images. To this end, we construct synthetic multi-object images by stitching the existing single-object-centric images, where the context of synthetic multi-object images can be predetermined. Therefore, our method can well address the random cropping issue on multi-object images.

• The comparison with MosRep

  Thanks for pointing out the missing references. We will carefully analyze and compare MosRep in the final revision.

  Compared to MosRep, the main contributions (differences) of our method can be summarized as follows.

  1. Our method provides an effective contrastive learning solution for multi-object images and achieves very competitive performance on both single-object-centric and multi-object tasks. In contrast, MosRep is only designed for the task on single-object-centric tasks.
  2. Our loss function is significantly different from MosRep. Our proposed contrastive objective models the accurate correspondences of multiple-to-single, multiple-to-multiple and single-to-single relations. The stitched images are addressed as a whole in our method. However, MosRep handles the mosaic representations as single objects by ROI Align.

  Moreover, we compare our method with MosRep according to the official implementation [R1] (The official source code of [R1] fails to achieve the results reported in the paper). The results are listed as follows.

  Method	#Views	Backbone	Linear	kNN
  MosRep [R1]	2×224+4×112	ResNet-50	72.3%	61.3%
  MOS (Ours)	2×224+4×112	ResNet-50	75.6%	70.6%

  The above results demonstrate that our method consistently outperforms MosRep by a large margin on both linear probing and kNN probing protocols.

  Moreover, we supplement the analysis of MosRep and cite it in the Related Work section.

  [R1] https://github.com/DerrickWang005/MosRep

• The refinement of the writing about method section

  Thanks for your constructive suggestion. We will refine these notations in the final version.

  For the notations in the paper, we mainly focus on the accurate expression of our method. We confirm that there is no useless notation or equation in our paper. In spite of somewhat complex notations, these notations can help authors to better understand the technique details of our method without confusion. Of course, we will further refine the writing of method section in the final revision.

• The application on multi-object images

  As clarified in the paper, our method is designed to simulate multi-object images by single-object-centric ones. Hence, the direct application of our method on multi-object images has not been considered in this paper. We would like to research this topic in future work.

Thank you for the response. I have carefully read the rebuttal and revision. However, my original concerns have not been addressed.

• More introduction about related works
  The main idea of this paper is to restrict the positive pairs of contrastive learning to avoid semantic inconsistency. This idea is highly relevant to the suggested [1-4], which still have not been discussed in the paper. These approaches belong to a different category than the region-level and pixel-level contrastive learning.

• The addressing of the issue about random cropping in multi-object images
  In essence, the paper avoids using multi-object images to sidestep the issue of semantic inconsistency. This approach largely overlooks the potential information that could be learned from such images, and bypasses the problem rather than addressing it. Given that the majority of real-world images contain multiple objects, how can one collect single-object images in a scalable manner?

• The comparison with MosRep
  Thank you for the additional experiments. The rebuttal claims that "Our method provides an effective contrastive learning solution for multi-object images and achieves very competitive performance on both single-object-centric and multi-object tasks. In contrast, MosRep is only designed for the task on single-object-centric tasks." However, I cannot find the experiments supporting this claim. Can the authors elaborate and demonstrate which parts of MOS make it effective for multi-object tasks, while MosRep cannot?

• The refinement of the writing about method section
  I did not claim that there are unnecessary notations or equations. However, providing excessive details, including those that are less essential, can distract the readers. For example, overloading equations with numerous superscripts and subscripts hampers readability. As one example, the paper uses subscripts $_i$ to indicate the $i$-th sample of the batch, but this $i$ is unnecessary when describing other parts; one can simply denote the sample as $x$ instead of $x_i$. The readers can still understand that the same approach can be extended to other samples in the batch. Delivering the core message in a readable form while maintaining precision is the art of mathematical writing.

We appreciate your thoughtful comments and resolve your main concerns as follows:

• Experiments in scenarios with highly dynamic or cluttered objects

  Thanks for your constructive suggestion.

  We evaluate our pretrained ViT-S/16 model (with FPN architecture) on WIDER FACE dataset for 80 epochs as [R1], where face objects present high degree of variability in scale, pose, expression, occlusion and illumination. The results (AP when IoU=0.5) are listed as follows.

  Method	Easy AP	Medium AP	Hard AP	mAP
  RetinaFace [R1]	96.9%	96.2%	91.9%	52.3
  MOS (ours)	97.3%	96.8%	92.8%	52.9

  Compared to the previous best method RetinaFace, our method achieves consistent performance gain on easy, medium and hard subsets of WIDER FACE dataset.

  [R1] RetinaFace: Single-stage Dense Face Localisation in the Wild. CVPR 2020.

• The comparative results about SelfPath pretrained for 300 epochs

  Thanks for pointing out this typo error. The actual number of pretraining epochs for SelfPatch is 300 epochs. We revise this typo error in the revised paper.

Thanks for your constructive comments. We address your main concerns as follows:

• The evidence for the performance degradation on multi-object images

  Thanks for pointing out the issue.

  The previous research works [R1, R2] indeed support that plain contrastive learning methods for single object-centric images suffer inferior performance on the multi-object images.

  For example, the introduction of [R1] claims that "for typical contrastive learning models, the pretraining pretext task considers an image holistically in instance discrimination, without explicit spatial modeling over regions. Though it enhances transferability for classification, this practice is less compatible with spatial reasoning tasks, such as object detection". This clarification is also validated in the experiments, such as Figure 1 of [R1].

  Moreover, our experimental results in Table 5 also demonstrate that our method achieves significant performance gain over the baseline pretrained by single object-centric images on both object detection and instance segmentation.

  We will supplement and emphasize this point in the final revision.

  [R1] Instance Localization for Self-supervised Detection Pretraining. CVPR 2021.

  [R2] Self-EMD: Self-Supervised Object Detection without ImageNet. Arxiv 2021.

• Why applying image stitching strategy on single-object images, such as ImageNet-1K and CIFAR?

  Although the images from both ImageNet-1K and CIFAR are single object-centric ones, we surprisingly find that our method consistently achieve very competitive results on image classification task for single object-centric images. Compared to [R1], our method also boosts the performance on image classification, instead of only spatial-sensitive tasks as [R1]. We speculate that our proposed stitching method introduces more image context information (include other objects) into training targets, thus alleviating the potential representation degradation (the plain methods may only focus on the most discriminative parts but miss other contents) and improving the generalization of representations.

• More analysis about the assumption on multi-object images

  Thanks for your constructive suggestion.

  We visualize the self-attention maps of our pretrained model on COCO dataset in Figure 11 (G The Visualization of Self-Attention Map section) and object correspondences between multi-object images with the same categories in Figure 12 (H Object Correspondence section).

  As shown in Figure 11, our method can well discriminate different objects in images. For example, the baby and teddy bear of the second row image about can be well distinguished from each others, yet the counterpart, iBOT, fails to discriminate them.

  In Figure 12, our method achieves rough matching between different multi-object images with the same categories. In the first image pair (both images contains dog and person), the correspondences successfully point out the semantic matching for the objects in the images.

  The above visualization results indeed demonstrate the effectiveness of our method on multi-object image representations.

• More ablation study on detection and segmentation tasks

  We supplement ablation results on object detection and instance segmentation tasks in Table 4 as follows.

  $L_{s2s}$	$L_{m2s}$	$L_{m2m}$	COCO AP$^{bb}$	COCO AP$^{mk}$
  ✓	--	--	42.8	38.9
  --	✓	--	43.1	38.8
  --	--	✓	NA	NA
  --	✓	✓	43.6	39.2
  ✓	✓	--	44.3	39.7
  ✓	✓	✓	45.6	40.6

  When the model is pretrained by all three losses $L_{s2s}$, $L_{m2s}$ and $L_{m2m}$, our method achieves the best results on both object detection and instance segmentation of COCO dataset. It indeed supports the effectiveness of the above three loss items. We supplement these results in the revised paper.

• More explanations about $L_{s2s}$

  The loss $L_{s2s}$ is just the normalized temperature-scaled cross entropy loss, which is widely adopted by the plain contrastive learning methods.

  We explain the design of the contrastive losses $L_{s2s}$ as follows.

  1. The patch boundaries of the stitched images adopted in our method are discontinuous, thus inevitably producing artificiality, which are different from the natural images. We call it domain gap between the stitched images and natural images.
  2. If we trained the model with only the stitched images, such as $L_{m2s}$ or $L_{m2m}$, the data for training and the ones for testing would be significantly different. Hence, the model would degrade the adaption of representations on natural images. The results in Table 4 support this clarification, where the training without $L_{s2s}$ obviously degrades the performance on downstream tasks.
  3. The application of $L_{s2s}$ can introduce natural images during pretraining, thus improving the representation adaption on natural images. It's also supported by the ablation results in Table 4.

• The performance on CNN

  As clarified in Introduction section, our method works better on vision transformer architectures. For CNNs, the artificiality produced by the boundary of image stitching may partially affect the representation performance. We evaluate our method on CNN architecture (ResNet50 pretrained for 200 epochs) as follows.

  Method	AP$^{\rm bb}$	AP$_{50}^{\rm bb}$	AP$_{75}^{\rm bb}$	AP$^{\rm mk}$	AP$_{50}^{\rm mk}$	AP$_{75}^{\rm mk}$
  DetCo (ResNet 50)	40.1	61.0	43.9	36.4	58.0	38.9
  MOS (ResNet 50)	41.2	62.4	44.7	37.3	59.2	39.5
  MOS (ViT-S/16)	45.6	65.9	48.9	40.6	62.8	42.6

  The results demonstrate that our method outperforms the previous state-of-the-art on CNN: DetCo, but worse than the ViT counterpart. It's a promising line to eliminate the gap caused by stitching artificiality. We plan to explore this topic in future work.