FlowFeat: Pixel-Dense Embedding of Motion Profiles

Thank you for your time and your valuable feedback. Please note that NeurIPS will provide an additional page for the camera-ready version. We plan to use the allocated space to expand our analysis of limitations and provide more intuition behind the approach. Thank you for these suggestions.

We clarify and respond to your comments below.

[...] optical flow is the natural task that should be evaluated.

Observe that FlowFeat is a monocular model – much like DINO and MAE. Since FlowFeat learns motion patterns from a pre-trained optical flow network, we do not expect -- and do not claim -- that FlowFeat will outperform the teacher network on flow estimation.

Instead, we focus on cross-task generalisation, as the common practice in representation learning. Therefore, we use benchmarks that are very different from optical flow estimation. Specifically, we evaluate FlowFeat on three benchmarks, train 9 model variants of FlowFeat and test 2 datasets for pre-training (YouTube-VOS and Kinetics). We believe that this evaluation scope and the consistency of the results across all experiments provides sufficient empirical evidence for our technical contribution.

[...] the paper lacks a deeper theoretical analysis of why motion profiles lead to better feature representations. For instance, how does the distribution of linear transformations generalize across tasks?

A theoretical analysis of our work is currently infeasible due to the fact that we are training a deep model on weakly curated video data with stochastic optimisation. However, the intuition behind generalisation of FlowFeat is the following.

The least-squares solution A* will be specific to the image pair, on which it was computed, and it will not generalise to fit the optical flow of another frame pair. However, by training the model on many frame pairs -- implying many different linear projections A* -- our embeddings learn to satisfy the induced distribution of those linear transformations w.r.t. the apparent motion. As a result, the network must encode non-trivial feature maps embedding plausible apparent motion.

[...] why does FlowFeat outperform [FeatUp]? Is it purely due to motion cues, or are there architectural differences?

This is a valid concern. However, we compare to FeatUp while ensuring the same backbone architecture. Therefore, the observed improvements come directly from the motion cues.

[...] How does FlowFeat perform on highly dynamic or occluded scenes? Are there failure modes not shown?

The VOS benchmark contains some highly dynamic and occluded scenes, and is broadly adopted in our community for these reasons. As shown quantitatively (c.f. Tab. 1), FlowFeat consistently improves the baseline accuracy.

As shown in some qualitative results, FlowFeat allocates less capacity to background elements. Quantitatively, the improvement on background areas is more modest in comparison to the improvement on the foreground regions. Nevertheless, FlowFeat demonstrates consistent benefits in terms of segmentation and depth metrics.

Can you provide more intuition or analysis for why motion profiles lead to better features? For example, is there a connection to equivariance or invariance in representation learning?

Intuitively, the field of apparent motion is highly structured -- recall the Gestalt principle “what moves together belongs together”. Now, let us consider constant motion as an example. In this case, approximating constant flow (no motion boundaries) is trivial both in terms of A* and the feature map. However, to approximate complex motion patterns with a linear projection A*, the network must learn a non-trivial feature map, which groups pixels that tend to have similar motion. As we train the network on a distribution of A*’s (induced by randomly sampling frame pairs), the network will learn a highly structured and spatially meaningful feature representation.

Indeed, FlowFeat is based on invariance of A* for overlapping image crops: the observed motion in one image crop must be sufficient to explain the motion in its adjacent crop.

Thank you for reading our response. As we did not recognise any critical concerns in the initial review, we hope for a more supportive recommendation.

Thank you for your thoughtful and encouraging feedback.

[...] the feature decoder does not receive a copy of the image.

We agree that this could be a valid concern. However, the DPT architecture, which FlowFeat uses, has skip connections to intermediate layers of the encoder, from which high-frequency information can be restored (despite the constant resolution in transformers). We kindly refer to Ranftl et al., [41] for architecture details and analysis. As can be seen from our qualitative examples (c.f. Figs. 1,3,4,5), FlowFeat exhibits fine-grained boundaries, suggesting no difficulty in recovering high-frequency information.

The authors make some non-obvious design choices, such as using an L1 loss instead of an L2 loss in Eq. 5 and the formulation of the weighting term in Eq. 6

We actually experiment with the two terms in our ablation study, please refer to Tab. 6, (c,d).

Since our training signal is an (potentially inaccurate) optical flow estimate, L1 distance is a natural choice due to its robustness in comparison to the L2 loss. We have verified this intuition by replacing the L1 training loss with the L2 loss. The results below, obtained on the VOS benchmark with DINO2-S14, confirm that L1 loss is a more appropriate choice.

[...] It would be very valuable to demonstrate that the presented findings also generalize to larger models commonly used for practical applications.

Please note that demonstrating consistent benefits across 5 commonly used backbone architectures already provides strong empirical support for our claims. Nevertheless, we have now experimented with larger models, DINOv2 ViT-L and MAE ViT-L, and report the results on VOS with linear probing. As expected, FlowFeat (pre-trained on YouTube-VOS) consistently and significantly improves the baselines, and surpasses the VOS accuracy of smaller models. These results suggest that FlowFeat also generalises to larger models.

Comparisons with FeatUp in Tab. 2 lack evaluation with the refinement stage for FeatUp and some base models, which should be included for a fair comparison.

Please note that FeatUp employs a stack of bilateral solvers; PAMR, at best, only approximates the bilateral solver with multiple local operations. Moreover, as can be seen in Fig. 1, FeatUp tends to produce feature maps with inaccurate or “smudgy” boundaries. The refinement with PAMR will correspondingly "smudge" the segmentation masks, if used with the FeatUp’s feature maps, degrading the segmentation accuracy.

As a remark, this is related to the earlier concern on high-frequency information. If FlowFeat lacked the high-frequency detail, the refinement would have no benefit. The results in Tab. 2 demonstrate the opposite.

Sec. 2 and 3 lack any structuring into subsections and/or paragraphs with headings [...]

Thank you for the great suggestion. We will improve readability of Sec. 2 and 3 by structuring the text into run-in headings.

Q.1 Regarding W1: Is this the case? [...]

As we discussed above – not exactly. The DPT decoder has skip connections, which allows FlowFeat to recover high-frequency detail, confirmed experimentally and qualitatively.

Q.2 What happens if the representations obtained from FlowFeat are downsampled to the same resolution as the base model's representations? [...]

This is an interesting idea. However, our experimental setting already accounts for the possibility of ‘mere’ upsampling of the feature representation in two ways.

First, we compare to FeatUp, which performs upscaling with a bilateral solver without computing a new representation. Intuitively, the FeatUp features remain in the ‘convex hull’ of the original representation available in the backbone. FlowFeat provides improved downstream accuracy on virtually all tasks (without refinement), which shows that the gains do not stem from the increased resolution alone.

Second, in the local KNN setting of VOS evaluation (see Tab. 1), we keep the same hyperparameters of label propagation (e.g. the size of the local window) across all models (c.f. ll. 50-54, supp. material). This requires us to downsample the feature maps of FlowFeat and FeatUp, which effectively implements the proposed setting. Here, FlowFeat also provides a tangibly higher accuracy across all baselines.

Overall, these results demonstrate that the gains come not merely from the increased resolution, but from the improved data representation as well.

Q.3 Regarding W2: Could the authors please elaborate on the motivation behind these choices?

As discussed above, the setting (d) in Tab. 3 already trains a model with the weight set to zero (i.e. no flow reconstruction loss). This leads to a slight decrease in the downstream accuracy on VOS.

Q.4 Are the representations directly related to the base encoder's representations in any way?

We are not sure about the question, but the FlowFeat representation comes from the DPT decoder, which has skip connections to the encoder model – this defines one relation. Moreover, FlowFeat is motion-driven, as it focuses on the foreground elements with large expected motion. By contrast, the base encoder’s representation is appearance-driven. Therefore, the two representations are different and complementary.

Q.5 Limitations are touched upon only very briefly and spread throughout the paper rather than being consolidated in a single place. [...]

Thank you for the great suggestion. We will consolidate the limitations in one place. Note that NeurIPS will provide an additional page for the camera-ready version, and we will be happy to use the allocated space to address this concern.

Q.5 [...] The PCA visualizations also appear to place a significant emphasis on moving objects [...]

This is an accurate observation and an expected outcome of motion parallax. However, our experimental protocol with semantic segmentation and depth estimation empirically validate that FlowFeat still preserves useful background information. We further elaborate on this point above in our response to reviewer meu9, to which we kindly refer.

Thank you for considering our response. We hope that it addresses your concerns and provides grounds for a more endorsing recommendation.

Thank you for your time and your valuable feedback.

We agree with both of your main points: the justification of the assumptions and the discussion of the limitations could be expanded upon, for which we had limited space in the submission. NeurIPS will provide an additional page for the camera-ready version, and we will be happy to use the allocated space to address your concerns, as we do below:

[...] the justification for this core assumption could be stronger.

The main idea behind the assumption derives from the following observation: the apparent motion (optical flow) is structured and spatially correlated. Therefore, the motion of two overlapping crops will be correlated as well. We model this correlation by a shared linear mapping A*, which predicts optical flow in both crops. As a result, the feature representation in both random crops will have a (partially) shared and spatially correlated structure.

Note that the intuition behind the cropping is not dissimilar to previous work (e.g. DINO), where two random crops are required to share a (global) representation. Our novelty is a dynamic computation of the global representation (A*), which additionally allows for efficient learning of per pixel motion-aware representations.

[...] how does it handle scenes with very little or no motion, or scenes with extremely complex, non-linear motion patterns?

We assume that the question relates to the training process, since FlowFeat is a monocular model (it takes only one image as the input).

When there is no motion, there is a trivial linear mapping (zero vector), so the learning gradient will be zero, effectively discarding the training sample.

State-of-the-art optical flow networks can handle relatively complex non-rigid motion, hence provide useful cues for training FlowFeat also in challenging non-rigid scenarios. Moreover, training FlowFeat is not very sensitive to errors in optical flow – thanks to the ridge regularisation in the least-squares solver (c.f. Eq. (4)) and the (robust) L1 training loss. However, we excluded compilation videos (i.e. multiple short clips stacked together) from the Kinetics dataset before training, since they break motion continuity.

The authors state that state-of-the-art networks, such as transformers, generate low-resolution feature grids that are suboptimal for dense prediction tasks. How is this suboptimality quantified or evaluated in practice?

All our experiments with the baselines are dense tasks, which exemplify one way of quantifying and evaluating the low-resolution feature grids. This setting corresponds to our baselines in Tabs. 1 and 2 (e.g. DINO, MAE).

Qualitatively, the limitations become particularly apparent in Figs. 1, 4 and 5.

[...] how do you construct a balanced training set for the linear classifier, especially for distinguishing the object from a diverse background?

We do not finetune or re-balance the training set for the linear probe. Indeed, the classification problem for fitting the linear probe in VOS can be highly imbalanced. However, this is not an issue, because the imbalance observed in the first frame may reflect the imbalance in the rest of the video and can therefore serve as a useful bias. A heuristic-free probing strategy is more transparent and makes the results more interpretable.

The ablation study in Table 3 shows that removing the L1 reconstruction loss (experiment d) leads to only a minor drop in accuracy [...]

This is because motion discontinuities are known to correlate well with semantic boundaries. Recall that we compute the gradient loss based on our lower-bound flow approximation with $A^\ast$ (c.f. Eq. (6)). Therefore, minimising the gradient loss with the solution $A^\ast$ will also lead to a lower reconstruction loss (c.f. Eq. (5)). This implies that the gradient loss jointly performs two functions: it focuses on motion discontinuities while minimising the reconstruction loss.

In practice, we observed that training FlowFeat with the gradient loss alone may be unstable for some backbone models and datasets -- presumably due to the spatial sparsity of the learning gradient (the loss vanishes for pixel areas with constant motion). Therefore, we keep both loss terms for consistency across all our experiments.

We hope that our response resolves your concerns and look forward to a more favourable recommendation.

While the authors noted that ‘in practice, we observed that training FlowFeat with the gradient loss alone may be unstable for some backbone models and datasets,’ a rigorous theoretical analysis would provide stronger support for this observation.

Thank you for your considerate, very encouraging feedback.

"[...] flow based features appear to lose background features when visualized [...]" / "[...] dive a bit more into the performance on "static background" elements [...]"

This is an accurate observation. Due to motion parallax, FlowFeat will prioritise foreground elements, where the apparent motion is expected to have a larger magnitude than that of the background pixels. However, FlowFeat also retains useful pixel-level information about the background structure. We show this empirically in Tab. 2 and elaborate further next.

Recall that NYUv2 is composed of static elements, which are semantically akin to background. On this benchmark, the improvements of FlowFeat over the baselines are consistent across all scenarios and notably surpass FeatUp. This suggests that FlowFeat not only preserves the static details (also visualised in Fig. 3, third row), but also leverages motion parallax as a useful depth cue.

Our segmentation results on COCO-Stuff are similarly consistent. As suggested, we further inspected the segmentation accuracy of non-dynamic categories. Specifically, we analysed the per-category IoU for semantic classes attributable to background areas (e.g. “trees”). As we report below, FlowFeat improves on those categories across all models, though somewhat less significantly than on (potentially) dynamic objects. For reference, we also provide the accuracy for ‘person’, as a typical foreground instance.

We will add analysis of this limitation to the final revision (e.g. in the additional 10th page). Thank you for the suggestion.

"[...] needing a good flow estimation network does limit it's applicability somewhat [...]"

Indeed, our approach requires a pre-trained optical flow network and video data. However, if the video sequences in another domain satisfy the assumption of brightness constancy, optical flow networks will likely still operate reliably. Consequently, a fine-tuned FlowFeat on the domain would also provide reasonable representations. However, we acknowledge that the success of FlowFeat ultimately depends on the training data available in the domain.

Q.1 How much feature map upsampling are you able to achieve with good results using FlowFeat?

As we show in the supplemental videos and Tab. 4 (supp.), scaling up the input to FlowFeat to resolution 448x448 improves the downstream accuracy on VOS further – in contrast to FeatUp. This resolution is already close to the upper bound of DAVIS (480p), which we used in the experiments. Following your suggestion, we have now inspected FlowFeat for the input resolution of 1080p. Since we cannot share any visual data, we report our observations. The PCA visualisation exhibits sharp boundaries, aligned with the image gradient and (potential) motion discontinuities. We observed no trace of artefacts akin to those emerging in FeatUp (c.f. supp. videos). Please note that evaluating transformers at resolution 1080p significantly increases their inference time and the memory footprint, thus we could not timely evaluate the models on benchmarks.

Q.2 Do the random crops used as input to the flow model have to be a specific resolution?

We feed frame pairs at resolution 224x224 to the flow network. Then, we sample the random crops from the produced flow map to align them with the input and output of the FlowFeat network. Therefore, the target flow maps used to train FlowFeat are bilinearly upsampled by a factor from the range (1, 2.5]. We did not find the training to be sensitive to moderate deviations from this range.

Q.3 Does the resolution of the flow network's outputs affect the DPT decoder's ability to upsample?

A low-resolution optical flow would conceivably result in smooth boundaries in the feature map of FlowFeat. Conversely, a high-resolution optical flow would increase the boundary sharpness, while also increasing the computational costs. We found a good compromise by running optical flow at the training resolution of FlowFeat (224x224) and upsampling the flow map to fit the input crops with bilinear interpolation, as described previously.

[...] the method only really applies to RGB web images, and not other domains [...]

We will point out the current application scope of FlowFeat to RGB images from the Internet – thank you for suggesting it.