MC-JEPA: A Joint-Embedding Predictive Architecture for Self-Supervised Learning of Motion and Content Features

Thank you for your comments and useful feedback. We address your concerns below:

The biggest weaknesses are in the experimental setup and missing comparisons with SoTA.

We incorporated a new experiment with a ConvNeXt-B backbone, as well as comparison with iBOT.

It is very hard to validate the performance of the proposed model when using a different backbone from the comparison methods. The only valid comparison is VICReg Vs. MC-JEPA because they both use CNX-T.

Our flow estimation method is incompatible with using a vision transformer. Indeed, the core principle is to first estimate a flow at a very coarse resolution using the deepest features of a convnet, and to progressively refine this flow by doubling the resolution at every layer, using the corresponding features that also double in resolution. The idea between ConvNeXt is to propose an architecture that is comparable in terms of size, number of parameters, throughput and performance, to recent vision transformer architecture. We therefore thought this would give a fair comparison with ViTs of the same size.

Many of the recent SSL models use ViT-S or ViT-B for performance evaluation. Results with ViT backbone would make the proposed model comparable to many other methods. At least, include ViT-S in Table 4 to compare with DINO and iBOT.

We propose a new experiment with a ConvNeXt-B backbone on linear frozen segmentation tasks, which we hope is fair against DINO and iBOT that were pre-trained with a ViT-B.

MC-JEPA pretrained with CNX-B is outperforming all the competitors on all the tasks except on ADE20k where it is on-par. A strong improvement over VICReg is still observed at this larger scale, which further demonstrates the effectiveness of motion learning for SSL.

Some methods, such as iBOT (ICLR22), are not reported. iBOT’s performance on ADE20k with ViT-S is 45.4, which outperforms MC-JEPA. Also if we compare against ViT-B of DINO or IBOT, MC-JEPA is outperformed on many benchmarks. It makes the paper stronger more relevant backbones and methods are included.

We will incorporate all our new results, as well as the iBOT performance in the main table.

Qualitative results of optical flow only shown against one relatively weaker method (as shown in Table 1). Results from SMURF and UPFlow could also be shown to visualize fail cases and limitations of this model.

As we mentioned in our answer to reviewer arP2, the goal of this paper is to demonstrate the usefulness of learning motion for SSL, which has never been demonstrated yet. Optical flow estimation is simply a pretext task and the goal is not to build a highly specialized flow estimator and produce SOTA flow estimation, but to produce high-quality features with a lot of semantic content. We refer to our answer to reviewer arP2 for a full justification.

This paper does not show any results of scaling up the model or training data; all backbones reported in table 4 are lightweight. I imagine that training two tasks like this would require a bigger backbone to handle multitasking. A study showing how performance scales with the model size would be useful.

We refer to our new result with a CNX-B backbone, which shows that the trend we observed at a lower scale is still valid with scaling.

The authors chose to do training on both tasks on very different datasets. I wonder if training this model on the same video dataset for both tasks would result in features robust enough for video downstream tasks, such as action recognition.

We agree that not training the motion and content objective on the same source dataset is the main limitation of this paper and we will focus on this goal for future work. The reason we did not tackle this setup first was that learning strong semantic features on video is still a challenge in the video SSL community, while learning strong features from image net is something well established. The goal of this paper was to test the hypothesis whether learning motion can help build better general SSL features, and our current setup and experiments validate this hypothesis which we will further explore in the future by learning from videos only.

Thank you for your comments and useful feedback. We address your concerns below:

The proposed method performs on par or slightly worse against other flow estimation methods on Sintel benchmark and Kitti, which can not demonstrate the benefit of the proposed ''Joint-Embedding Predictive Architecture''. The authors don't give convincing analysis for this issue.

Contrary to other flow estimation methods, which have the objective of producing the best quality flow possible, our goal is to leverage flow estimation as a pretext task in order to learn self-supervised representations from images. We therefore choose to keep the flow estimation method simple, and to only make the necessary adjustments that we describe in our answer to reviewer q59r to make it work in a multi-task setup with regular self-supervised learning of semantic content features. We compare the quality of our produced flow to show that it is of comparable quality than popular unsupervised flow estimation methods, without using a complex list of tricks.

Unfair comparison. The proposed method brings more training data compared with the other content methods. It can be seen from Table 2 that the performance in segmentation is likely to degradation as the decrease of motion datasets.

The other content methods are only trained on ImageNet, which has no motion information. The goal of this paper is to demonstrate the usefulness of learning motion for SSL, and thus bringing optical flow estimation from video datasets, and this is demonstrated by comparing MC-JEPA to VICReg and showing strong improvement on segmentation tasks. The ablation of Table 2 further shows that regardless of the specific flow estimation dataset, a strong improvement is still observed. Indeed for example on Pascal VOC linear frozen segmentation, VICReg performs 60.1 mIoU. MC-JEPA variants with different flow datasets perform between 66.8 and 67.1 which is a very strong improvement, with a very low variance when the flow dataset changes.

Missing analysis. From Table 4, we can see that the model's performance is quite sensitive to the used backbone (more than 10% between Rsenet50 and ConvNext. However, the authors didn't give explanation. Besides, the proposed model uses six loss terms in total for both flow estimation and content learning. I am wondering how to decide the trade-offs and if they would affect the final results.

In Table 4 we compare 3 backbones, PWC-Net, ResNet-50 and ConvNeXt-T. For flow estimation all these backbones perform similarly. However, for semantic content learning, PWC-Net, which is a very shallow network specialized for flow estimation, and it therefore does not perform well on segmentation downstream tasks. The gap in performance between ResNet-50 and ConvNeXt-T is not something specific to this paper, and is observed even in supervised learning on ImageNet. ConvNeXt is an architecture that was designed to perform as good as modern vision transformers and it is not surprising to see such gaps for segmentation tasks. We indeed choose ConvNeXt over ResNet from the beginning for this specific reason.

We refer to the explanation we gave to reviewer q59r on how we tune the coefficients of our losses.

Thank you for your comments and useful feedback. We address your concerns below:

The writing of the paper needs to further improve. The captions of the figures and tables are too detailed. It is better to condense the captions.

The objective was to have self-contained captions, which could be fully understood without diving into the main text. We will change the approach and simplify the captions.

In multi-task learning, how are the weights for different loss functions chosen and adjusted? Will adjusting coefficients for the six different loss functions for each task introduce a significant tuning cost during training? Would it be possible to draw comparisons between this tuning approach and other methodologies in the field of multi-task learning?

We refer to the explanation from reviewer q59r on how we tune the coefficients of our losses. Our approach is simple and does not require a lot of tuning.

2.Although some experiments have already been conducted to demonstrate improvements in Optical Flow Estimation, Image Segmentation, and Video Segmentation, additional experiments could be included to further validate the effectiveness of the proposed methods. For example, more experiments on video object segmentation (YoutubeVOS), video semantic segmentation, video panoptic segmentation(Cityscapes-VPS ,VIPSeg ).

We decided to focus on linear frozen segmentation (Pascal, Cityscapes, ADE20k) and non-parametric evaluation (DAVIS 2017), which we believe are more suited to evaluate SSL representations. We nevertheless provide new experiments were we fine-tune our model on YoutubeVOS and Cityscapes-VPS:

Our results further demonstrate the superiority of the flow estimation multi-task over regular semantic content learning.

What's the motivation of the multi-task learning ? Learning optical flow seems to be a low level visual understanding, why it benefits semantic segmentation?

Optical flow estimation is indeed a low-level task, and the goal of this paper is to test the following hypothesis: can a low-level task help ground self-supervised training to help learn higher quality high-level representations ? To our surprise, the answer is yes, which we show by comparing MC-JEPA with VICReg and demonstrating strong improvements across all segmentations downstream tasks. Our intuition is that optical flow estimation in this multi-task setup helps structuring the representations with a locality bias, and observing with which pixel is moving where, helps understanding to which object this pixel belongs.

Thank you for your comments and useful feedback. We address your concerns below:

1) Novelty. The main novelty of our paper is more about the setup (multi-task, flow estimation and semantic content learning), and the discovery that flow estimation helps segmentation downstream tasks in SSL, rather than the specific architectural details. However we list here the different new components that we introduced in the paper:

Architecture:

• Adaptation of the PWC-Net general principle to work with a custom ConvNeXt backbone. We start with the PWC-Net architecture which uses a specific backbone that we replace with a ConvNeXt backbone. This replacement is necessary to make the content learning part work, and requires to replace the first convolutional layers of ConvNeXt by custom layers in order to make the resolution of the feature maps double at each new level. The replacements are described in Appendix C.

• Flow estimator network architecture. We start from the PWC-Net flow estimator and regularize its output with a normalization layer. This is an essential component for the multi-task setup and the flow estimation network is very unstable without it. In addition we increase the number of convolutional filters and number of layers, which shows a positive scaling trend and improves the quality of the produced flow. We present an ablation on network size normalization method in Table 3 and give more details in Appendix C.

Loss functions:

• New cycle consistency loss. We introduce a new cycle consistency loss that forces f(f^{-1}(x)) = x, where f is the flow and x is the feature map. This loss has a positive impact on the flow estimation quality, which is studied in Figure 5 (2), and is introduced in this paper for the first time.

• New regression and regularization losses. The new multi-task setup we propose was unstable at first and required the addition of flow regression losses at every layer, as well as variance covariance regularization losses. We are the first to make this setup work and this an improvement over regular SSL methods such as VICReg which applies the regularization only at the last layer, and flow estimation methods which apply a reconstruction loss only at the last layer as well.

In addition to these architectural novelties, as we mentioned we focus on the setup and bring various new elements such a study on different multi-task setups in Table 5 and a study on training with different datasets in Table 2.

2) Loss coefficient tuning. The MC-JEPA method consists in two tasks: a flow estimation task that learns motion and a SSL general task that learns semantic content. Figure 5 (3) shows how we adjust the coefficient balancing these two tasks. The caption was not clear as it refers to Eq.7 which has no coefficient. We will update Eq.7 with a balancing coefficient $\alpha$ depicted in Figure 5 (3). In our experiments we test several values that are all depicted in the figure, and take the best value, $\alpha=0.1$, which gives the best performance for both flow estimation and segmentation. The flow estimation task is further divided into several loss functions. These loss functions are applied at different levels of the convolutional backbone, and we balance these losses using different sets of coefficients for the different levels. The general idea is to decrease the value of the coefficients as we go deeper in the backbone, in order to focus more on flow reconstruction at the highest resolutions, while still enforcing a coherent flow at the lowest resolutions.

At the first layer, we fix all these coefficients to 1.0, except for the variance loss coefficient which we fix to 25.0 following the VICReg paper. For the deeper layers, we progressively decrease the coefficients of all the flow losses together, and decrease variance and covariance regularization coefficients even further as these are less and less required to maintain stability. In our experiments, we found that the specific values of these coefficients do not matter as long as the general principles we described are followed.