Is ImageNet worth 1 video? Learning strong image encoders from 1 long unlabelled video

This paper investigates self-supervised representation learning from video with a specific focus on the data distributions. In particular, the paper questions the needs of using large-internet scale image datasets and propose instead to learn representation by watching few long videos.

The paper makes two mains contributions:

• The WTtour datasets which composed by 10 long-videos
• DORA, a self-supervised representation learning approach that learns to represent and track object in a video at the same time.

The paper evaluates the learned representations on various downstream tasks including ImageNet linear probing, Pascal VOC unsupervised object discovery, MS-COCO/ADE20K object detection/segmentation and DAVIS-2017 for video understanding. DORA pretrained on WTours demonstrates strong performances on ADE20K and MS-COCO

This work introduces a method for self-supervised learning of image representation models. It is based on a similar scheme to DINO (Caron et. al. 2021) that learns image representation by distilling a moving average teacher model's representation of the global image view to the representation of a student model's on multiple local views. The novel way of achieving this in this work is to use the feature correspondence provided in hours-long walking tour videos to provide tracking, which can identify the location of different objects in any video frame without annotation. This object-centric way of generating local views seems to lead to good learned representation, which is examined in the experimental section.

In total, the proposed method does not need curated image or video data for self-supervised learning and can learn effective representation from hours-long walking tour videos. The proposed multi-object masking approach is shown to be significant in learning effective visual representation.

This paper proposed a new perspective on self-supervised learning (SSL). Instead of pretraining models on ImageNet-like object-centric datasets, the paper pretrained the models on egocentric videos (“Walking Tours” dataset) which depict numbers of objects and are comparable with human learning. Compared with other video datasets for SSL, Walking Tour dataset had more objects and classes and more gradual shifts in lightness. To pretrain on Walking Tours, the paper proposed a novel SSL method, based on DINO, to first discover objects and then track objects, named DoRA. In every batch, DoRA randomly sampled 8 frames temporally separated by 1 second, discovered objects in the first frame, and tracked them over the following 7 frames. In the default setting, objects are tracked by cross-attention in the multi-object tracker, which leads to spatially overlapping. The paper then proposed to establish object-patch correspondences using the Sinkhorn-Knopp algorithm to deal with the problem. After finding separated objects, the input video clip in the student branch would be masked to contrast with the clip in the teacher branch. Experiments on dense prediction tasks show that DoRA on Walking tours shows comparable performance with other SSL methods pretrained on ImageNet.

The paper proposed a dataset Walking Tours (WT) consisting of high definition street-walk videos of 23 hours total length, and DoRA, a multi-object-tracking-inspired framework to learn visual representation from different views of the same objects in adjacent frames. The proposed method largely follows the DINO framework, with the local crops in DINO replaced with a tracking of objects in a video. The method is tested on several mainstream visual tasks, including image detection and segmentation, video object segmentation, object tracking, image classification and object discovery. Compared to its baseline DINO, the proposed method outperforms it clearly when using the same WT dataset.

The authors consider how both (a) data and (b) method can improve training an image encoder in a self-supervised manner. Regarding data, they introduce an open-source dataset containing long, first-person videos, and propose several advantages of this over curated image (and video) datasets. Given this dataset contribution, the authors propose a self-supervised method which tracks objects to act as a signal for a classical multi-view self-supervised learning (SSL) loss. This method leverages some key properties from the dataset, specifically that the videos have natural scene transitions and are of a person-person view. Instead of using off-the-shelf object-trackers or optical flow to establish correspondence, the authors use the attention-map between the [CLS] token from a selection of heads and the patch-embeddings. Optimal transport is used to establish unique object-patch correspondence (i.e. non overlapping patches) and then given this multi-view correspondence, a technique based on DINO is used.

Overall the authors show that for many downstream tasks, their method (DORA) pre-trained on (even one) video where the number of frames is comparable to imagenet-1K (IN-1K) achieves better performance than DINO pretrained on IN-1k. In comparison when DINO is pretrained on the same data, the peformance is worse than IN-1K which suggests it is the unique coupling of dataset and training-method which helps the authors achieve SOTA results

Indeed, DoRA (WT-all) is on-par with DoRA (WT-Venice) on linear probing on ImNet. However, in terms of $k$-NN on ImNet, DoRA (WT-all) outperforms DoRA (WT-Venice) by 2%. In ``Common response to R-DfLs, R-CACu, R-GQ4C: Results on ImageNet linear probing", we show that DoRA (WT-all) outperforms DoRA (WT-Venice) and DINO (ImNet) when fine-tuned on 5 classification tasks. We also observe in Table 3 that DoRA gives superior performance when fully fine-tuned on spatially dense tasks, similar to MAE.

In Table 4 (object tracking), DoRA (WT-all) outperforms DoRA (WT-Venice) by 4.5% in mAO, and 5.5% in SR$_{0.75}$. Additionally, in Table 3 (semantic segmentation on ADE20K), we observe that DoRA (WT-all) outperforms DoRA (WT-Venice) by 1.5% in mIoU and 2.5% in terms of Acc$_m$. In the same table (instance segmentation on MS-COCO), DoRA (WT-all) is better by 1.6%.

The drop in performance of DoRA (WT-all) with respect to DoRA (WT-Venice) in Table 4 (video object segmentation) refers only to a single downstream task out of seven in the paper.

Finally, we have added more results in Section C under ``Longer pretraining" and Table 7 in the Appendix. These new results show that, when pretrained for 300 epochs (instead of 100 epochs), DoRA (WT-all) significantly outperforms DoRA (WT-Venice) on image based downstream tasks.

We evaluate DoRA (WT-Venice) for 100 epochs on ViT-B/16 for semantic segmentation on ADE20K and object detection on MS-COCO. The results are summarized in the Table below.

We observe that with ViT-B/16, DoRA achieves 4.9% gain in terms of mIoU on ADE20K and 2.2% in terms of mAP on MS-COCO as compared to using ViT-S/16. This shows that DoRA also scales well with the model size. We thank the reviewers for this suggestion. We shall add the results with ViT-B/16 for all downstream tasks.

As R-DfLs and R-GQ4C point out, because of the domain gap between WT dataset and ImageNet, there is a large gap for linear probing on ImageNet itself. As per the reviewers suggestion, we follow the evaluation protocol in iBOT and finetune DINO (ImNet), DoRA (WT-Venice) and DoRA (WT-all) using ViT-S/16 on CIFAR-10/100 (C-10,C-100), iNaturalist18 (iNat 18), Oxford Flowers (Flwrs), Stanford Cars (Cars) and ImageNet-1k (ImNet) dataset. The results are given in the table below:

Despite the class distribution of iNat18 being closer to ImNet than WTours, we observe that DoRA (WT-Venice) is on-par with DINO (ImNet) when fine-tuned on C-10/C-100, iNat18, Flowers and Cars datasets. Furthermore, DoRA (WT-all) outperforms DINO on ImNet on all fine-tuning tasks. We also observe such performance gains when finetuned on semantic segmentation on ADE20K and object detection on MS-COCO (Table 3 of main paper), similar to MAE.

Additionally, we also perform scene classification on Places205, measuring classification accuracy, using linear probing on models pre-trained on WT-Venice and WT-all for 100 epochs. The results are in the table below.

These results indicate that the gap on linear probing on Imagenet is due to the domain gap and not a gap between images and video. We thank the reviewers for this very interesting comment. We shall add these results.

We thank the reviewers for this interesting comment. As per their suggestion, we compute the training throughput and the overall training training time of DoRA (WT-Venice) and DINO (WT-Venice). On the NVIDIA A100-80GB GPU, DINO achieves an training throughput of 1907 im/sec and DoRA achieves 1598 im/sec, averaged over 10 runs. For a single epoch, DINO uses 4hr and 5min of pretraining time on 8 A100-80GB GPUs, while DoRA uses 6 hr and 48 min.

To compensate for the increase in training time, we compare DoRA pretrained for 60 epochs with DINO pretrained for 100 epochs on ImNet in Appendix D and Table 8. We show that the improvement in performance is due to our multi-object masking.

The computational overhead due to SK is minimal as DoRA uses only 30 SK iterations. The throughput of DoRA (without SK) is 1883 im/sec, while DoRA (with SK) is 1860 im/sec.

The paper presents a self-supervised setting on egocentric videos and introduces an egocentric video dataset to demonstrate its use cases. Moreover, the paper presents a novel method to learn from continuous videos based on "tracking to learn to recognize" approach.

After taking into account the five reviews and discussions among reviewers, the strengths and weaknesses are summarized below:

Strengths summarized below:

1. the paper is well-written and organized; the problem is well-motivated and less studied in the literature
2. the data and source code will be made available, useful for reproducibility
3. contribution of a new egocentric video dataset with distinct features from other existing datasets
4. contribution of a novel method leveraging on "tracking to learn to recognize".
5. results in several down-stream tasks are impressive

Weakness summarized below:

1. ethical concerns due to the new video dataset contributions
2. slightly worse results on image classification, such as on imagenet datasets, ADE20K, and MSCOCO tasks; justification about lower performance is needed
3. more explorations are needed to test what type of egocentric videos are suitable for DoRA (the proposed model) or general self-supervised learning methods on videos

Accept (oral)