PooDLe🐩: Pooled and dense self-supervised learning from naturalistic videos

Thank you for taking the time to review our work and providing valuable feedback. We appreciate the positive comments on our method’s technical soundness, demonstrated superior performance on dense prediction tasks, and justifying ablations on PooDLe’s design components and on pretraining data attributes. We would like to address your concerns and questions about the evaluation below.

i) About SDM

We acknowledge that the SDM adds parameters and computation to a ResNet50. In comparison, PixPro [1] utilizes a similar FPN module, which we use in our evaluations. Furthermore, for other ResNet-50 baselines, we follow DeepLab [2] for evaluation, which uses dilated convolutions on the last 2 ResNet stages to increase the resolution of feature maps – this setup improves performance. FlowE also uses the same dilated setup for pretraining. While dilations add no parameters, it significantly increases FLOPs. The SDM is an efficient replacement for dilations in both performance and compute cost; we provide the FLOPs of different inference architectures for a single 512x1024 image below.

Also note: the SDM (frozen weights) and dilations are kept during linear readout semseg, but are removed in UperNet semseg and object detection, where PooDLe also achieves the best results. This also suggests that training with the SDM learns a stronger ResNet50 representation than dilations, even when it is removed at inference.

Table A. Inference overhead of different model architectures

ii) Some comparisons are apples-to-oranges

You are correct – we will revise our claims to reflect the differences in between the methods, as we did not include ViT experiments. However, we would like to point out that even with the differences, in L270-282, PooDLe shows a notable improvement over DoRA on BDD and CityScapes semantic segmentation, e.g., +6.5 mIoU on BDD UperNet semantic segmentation.*

• For completeness, on BDD, DoRA trained with 100 epochs attains 40.8 mIoU on BDD UperNet semantic segmentation versus 43.3 mIoU for 200 epochs of training.

iii) Tasks beyond dense prediction

This question brings up an interesting point. iBOT [3] demonstrated that augmenting global supervision with a dense objective improves image classification, but this work only trained on iconic ImageNet data. We find that iBOT and other global methods do not transfer well to naturalistic data composed of dense scenes, varying objects and imbalanced classes. However, our work is towards learning better representations from this naturalistic data; first for dense prediction tasks, but eventually for semantic global tasks as well. We also note that most naturalistic datasets, including BDD and Walking Tours, do not have global prediction tasks.

iv) About the "relationship between global crop area and input resolution"

Thank you for pointing this out – we do not directly explain the relationship between these two factors. To clarify, while developing PooDLe, we noticed that performance for higher resolutions would peak at larger global crop areas, i.e., larger fields of view. This is shown in Figure 5 where performance for 512x1024 is maximized at 0.275 mean crop area while 256x512 is maximized at 0.125. Thus, PooDLe seems to perform best at a specific pixel-scale, or a pixel density per ‘feature’ in the dense feature map.

We will revise Contribution 3 to better reflect what is shown in Figure 5 and clarify the message of the accompanying paragraph in Section 4.4.

v) Patch size

We used a patch size of 16 for the ViT-S baselines.

vi) Walking Tours videos

Yes, the original videos of Walking Tours are at 4K and 60 fps. However, we note that the dataset’s download code only supports 720p and DoRA’s implementation details stated that 30 fps was used [4]. We can revise and mention these details in our paper.

[1] Propagate Yourself: Exploring Pixel-Level Consistency for Unsupervised Visual Representation Learning. CVPR 2021.
[2] DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs. TPAMI 2017.
[3] iBOT: Image BERT Pre-Training with Online Tokenizer. ICLR 2022.
[4] Is ImageNet worth 1 video? Learning strong image encoders from 1 long unlabelled video. ICLR 2024.

Thank you for taking the time to review our work and provide useful feedback. We appreciate the positive comments on our paper’s clear writing, the motivation for our method, and the thoroughness of our experiments for demonstrating PooDLe’s effectiveness across different settings.

We would like to address your comments and questions below.

1. Relationship between SDM and Pooled objective + ablation experiments (weakness #1)

Adding the pooled objective alone does not improve performance because the same representation has to satisfy two different objectives. The pooled objective trains the encoder features to capture semantic information over small local crops. The dense objective focuses on scene-level understanding and without the SDM, it largely ignores small objects: 8.7 => 12.8 mIoU, row 1 to row 5 in Table 4. However, having the SDM enables the model to propagate the high-level semantic information learned by the pooled objective and upsample it to higher-resolution features used in the dense objective.

The extra parameters added by the SDM are top-down ResNet blocks, which are shown in row 3 of Table 4 and do not provide improvement on small objects. Only when the lateral connections (single, linear projections) to low-level features are added do we see some improvement, as shown from row 1 to row 5: 8.7 => 12.8 mIoU. Finally, when we combine the SDM with the pooled objective, we observe further improvement from row 5 to row 7: 12.8 => 15.0 mIoU.

Note on technical contributions (weakness #2)

The technical contributions of our work include the integration of dense and pooled learning objectives alongside the SDM. As discussed above, our ablations show that only when we combine all three components do we see the most improvement on learning from dense scenes and overcoming spatial imbalance in naturalistic data.

2. Figure 4 tradeoff between small and large objects

The goal of Figure 4 is to show that FlowE misses small objects like the people on the far sidewalk, and that baselines which use pooled, iconic-based objectives produce noisy object boundaries. While PooDLe produces a noisier outline of the front car compared to FlowE, it also predicts a cleaner boundary for the buildings and the car on the right. Overall, in our experiments, we find that PooDLe still outperforms FlowE on large objects by 51.4 to 49.3 mIoU (Table 3).

3. Improved small object capture with random subcrops

This is a great question. While the random crop sampling doesn’t increase the probability of viewing any one pixel of a small object, it does increase the probability that some sizable part of the object is seen. Coupled with the reduced likelihood of capturing multiple objects in each subcrop, we can apply our pooled objective to learn stronger object semantics by using each subcrop as a ‘pseudo-iconic’ example.

10. Training overhead of additional subcrops

Thank you for noting this! The subcrops are at very small resolutions compared to our global crops and do not contribute much to training cost. We compare the training overhead of PooDLe with 6 subcrop pairs and without subcrops on a RTX A6000 with batch_size=4.

Table D. Training overhead of subcrops and pooled objective for PooDLe

11. Additional delta_t values for Figure 9

We provide the BDD linear semseg results for delta_t=8 and the results from Figure 9 for reference. Delta_t=8 also performs well, but delta_t=15 appears to be the sweet spot for sufficient and not too noisy video motion for learning.

Table E. BDD linear readout semantic segmentation for varying delta_t

12. Predictor and projector

We refer to BYOL [1] for the introduction and analysis around the use of projector and predictor modules, as they are first introduced in that work. We omitted them in our method figure for clarity, as they are often omitted in other method figures, e.g. DINO [2].

13. The use of global pooling on subcrops

Our work aims to mitigate the spatial imbalance problem of fine-grained objectives which prioritize larger background classes by introducing a pooled learning objective over smaller subcrops. If we do not pool the subcrop features, then the subcrop objective (L205) would suffer from the same spatial imbalance problem as the dense objective. Furthermore, the subcrops would likely not perform well for the dense objective because they are designed to capture different views of single objects.

[1] Bootstrap your own latent: A new approach to self-supervised Learning. NeurIPS 2020.

[2] Emerging Properties in Self-Supervised Vision Transformers. ICCV 2021.

4. Comprehensiveness of ablation experiments

We are happy to provide the accuracy metric for our ablation table – we had omitted it for brevity and in favor of the Small/Large, Rare/Common mIoU breakdown. Note that the small/large and rare/common accuracy breakdowns are averaged over the class subsets, while the ‘All’ category is over pixels. Here are the accuracy values for our ablations in Table 3 and Table 4, which we will also add to the Appendix:

Table A. Accuracy values for Table 3 with breakdown by class grouping.

*Pretrained on BDD, initialized with supervised IN1K weights.

Table B. Accuracy values for ablation Table 4 with class grouping.

†Flow model trained without supervised labels.

We also believe it is standard practice to run ablations on only one dataset to demonstrate the effect of our method contributions of dense and pooled losses and the SDM module. BDD is a large and diverse dataset, featuring different cities, time of day, and rates of ego-motion, as the car can be moving or standing still. We show the effectiveness of our method on other datasets by training and evaluating on the egocentric Walking Tours videos.

5. Learning with only the pooled loss

We actually did run this ablation, which is equivalent to applying the pooled objective to many small crops captured obtained throughout the frame. However, it performed very poorly, with 26.59% mIoU at the ablation scale – far worse than other methods.

6. FlowE results on Walking Tours

We were not able to run FlowE in the full setting on Walking Tours due to computational constraints, but we are able to provide results in an ablation scale setting, using 192x384 resolution global crops and 20 epochs of training. The table below shows that PooDLe outperforms FlowE on ADE20K linear semseg.

Table C. ADE20K linear readout semantic segmentation, pre-training on Walking Tours

7. ViT for PooDLe

We demonstrated the contributions of our method using ResNet-50, and we generally expect them to transfer to ViTs. However, some further work is likely necessary as there is no resolution hierarchy in ViTs which would affect the SDM’s design and the flow equivariance (dense) objective has not been used with ViTs previously. We leave these extensions for future work.

8. Would this method work for classification?

We believe PooDLe representations would perform reasonably well on classification, as our models learn semantic information via the pooled objective and are capable of differentiating between classes for semantic segmentation, i.e., pixel-level classification. Nonetheless, there are no global classification benchmarks associated with BDD or Walking Tours. However, please note that the goal of our work is to improve representation learning on naturalistic, uncurated data. This is challenging in comparison to learning from global classification datasets like ImageNet that are curated to be iconic and class-balanced.

9. Accuracy values on Table 3 & 4

We provide accuracy values for these tables in #4 above.

Note on accuracy improvement (weakness #3)

PooDLe’s accuracy improvement from 88.5 to 89.2 over FlowE is significant at such high accuracies and represents a 6% reduction in error rate. In addition, as the classes are highly imbalanced, incremental improvements are very difficult. Conversely, mIoU weighs each class equally and is usually the primary metric reported in prior works. Additionally, at a high level, it is more important to capture all classes well than to focus on per-pixel accuracy.

Thank you for spending the time to review our work and provide insightful comments. We are encouraged that you found our problem area to be well-motivated and our approach to be straightforward and sensible. We would like to address your comments and questions on cropping below.

First, we would like to make a general clarification on how we apply the two objectives to the image crops. For each pair of video frames, we crop a pair of larger global crops and then K pairs of smaller local subcrops from the global crop pair. We compute the dense loss from representations of the global crops, as they have a larger field of view with many objects. We compute the pooled loss from the subcrops as they are much more likely to contain a single object, allowing pooling and invariance-based learning.

Smarter choice of subcrops

This is a good observation. We are actually currently experimenting with saliency methods like ContrastiveCrop [1] and outlier methods like SSD [2] to improve crop selection. We had also considered using flow to find fast or independently moving objects, or flow clustering [3]. However, we did not continue this as objects of interest are not always moving, and camera ego-motion may make things appear to be moving when they are not. For instance, a bike may be locked to a fence and stationary, while cars are large, common moving objects that are already easy to capture. Additionally, these are observations specific to driving data like BDD and may not apply if we are in another setting with mostly static objects (such as a museum). Nonetheless, random small subcrops still improve foreground object hit-rate, as you mentioned and as analyzed in Section 4.5 of our paper.

Changing the number of subcrops in Figure 7

Following our clarification above, it is varying K, the number of paired subcrops generated from the pair of global crops. This increases the number of terms averaged for the loss in Eq. 2. For 0 subcrops, we do not use the pooled objective at all.

Flow-matching for subcrop selection

We are running two ablations to measure the importance of flow-matching for subcrops: (1) selecting 2 random, independent subcrops and (2) selecting 2 subcrops at the same location from the pair of global crops. It’s expected that (2) will perform better, but will be affected by the data and the amount of motion between frames. To quantify this effect, we are also running (2) at the default delta_t=15 as well as delta_t=45. We will provide an update when we have these results.

Table 4 and subcrop resolution

We’ll answer these questions together as they’re related to our general clarification above. FlowE is only trained with a dense objective and thus only on the global crop pair. Subcrops are used for the pooled objective in Table 4. During training for the main results, the same model backbone encodes subcrops at 192x192 resolution and the global crop at 512x1024 resolution. Ablation experiments use 192x192 subcrops and 256x512 global crops. We evaluate at 512x1024 for BDD100K and 512x512 for ADE20K tasks for both the main and ablation experiments, indicating that the model can generalize to different test time resolutions.

[1] Crafting Better Contrastive Views for Siamese Representation Learning. CVPR 2022.

[2] SSD: A Unified Framework for Self-Supervised Outlier Detection. ICLR 2021.

[3] Guess What Moves: Unsupervised Video and Image Segmentation by Anticipating Motion. BMVC 2022.

Below are the results for the flow-matching ablations. We find that selecting subcrops at the same location in an image, with jitter (2) slightly outperforms flow-matching (1) for BDD pretraining at delta_t=15, but degrades quickly as delta_t increases to 45, obtaining the same mIoU and lower accuracy (-0.4%). Unsurprisingly, independent random crops performs very poorly, as the subcrops are very unlikely to contain the same subject.

This is a matter of hyperparameter selection, as the benefit of flow-matching for subcrop selection is dependent on object motion and object size. In particular, flow-matching would be more helpful for large, faster moving objects and should also benefit from more targeted subcrop strategies.

Table A: Comparison of different subcrop selection methods at dt=15,30,45

Thank you for spending the time to review our work. We appreciate the positive feedback, and suggested points of clarification on computation cost and sensitivity to video properties.

We will address your comments and questions below.

Optical flow estimation cost

First, we note that optical flow estimation is not used in real-time inference – it is only used in pre-training. It takes 0.069 seconds, 18.9% of a training iteration, to estimate optical flow using RAFT (24 iterations) for a pair of 512x1024 images. This is further diminished during training due to PyTorch optimizations. Additionally, our method should perform comparably with fewer flow iterations and the optical flow results can also be computed once and stored on disk.

Computational overhead of the SDM compared to simpler upsampling approaches

Without the SDM, prior methods use dilated convolutions for upsampling features [1, 2, 3]. We compare the FLOPs of a ResNet-50 + SDM vs ResNet50 + dilated convolutions for 4x upsampling on one 512x1024 image. Our SDM is more efficient by FLOPs. The SDM also improves memory usage in training, as seen in the next table below.

Table A. Computational overhead of upsampling approaches

Analysis of inference time and GPU memory requirements

We use a standard ResNet-50 as our encoder and ResNet blocks in our SDM, so we expect our model’s inference time scaling to follow most vision encoders.

GPU memory requirements compared to baselines

Here is a comparison of the training VRAM requirements of PooDLe compared to representative baselines with one sample. For PooDLe, this count includes the joint pooled and dense objectives. In comparison to the SDM, the dilated convolutions in FlowE quadratically increase the activation count in the 3rd and 4th ResNet stages, leading to greater VRAM usage.

Table B. GPU memory requirements of different methods

Sensitivity to video frame rate or motion blur

In Figure 9, we demonstrate that our method maintains strong performance across different temporal stride sampling, which is analogous to varying video frame rate. Motion blur is difficult to directly quantify, but we note that our method is able to learn useful representations despite the wide range of motion levels in BDD100K and Walking Tours.

[1] DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs. TPAMI 2017.

[2] Pyramid Scene Parsing Network. CVPR 2017.

[3] Self-Supervised Representation Learning from Flow Equivariance. ICCV 2021.