Moving Off-the-Grid: Scene-Grounded Video Representations

Thank you for your detailed review and constructive feedback. We were pleased to hear that you consider our contribution to be novel, general, and the ideas widely applicable. We appreciate that you recognize the strength of our qualitative experiments to help understand the method’s working.

“I had a hard time understanding the point of the corrector for a while. [...] the predictor is just a way to establish alignment between features in neighboring frames so that the cross attention of the corrector is more effective?”

Thank you for pointing out this confusion. We will clarify this in the updated version of the paper. Indeed, as is mentioned in the paper, the separation between the corrector and predictor is slightly artificial if the predictor is not unrolled in an open-loop prediction set-up. Since this is a deterministic model, unrolling in an open loop would produce blurry predictions quite quickly, but this indeed an interesting future direction.

The purpose of the corrector is to integrate new information in the state based on the current observation. However, if we were to decode from this state it could create a shortcut that the model could exploit (the model could learn to ignore the temporal prediction component, i.e. the alignment between features in neighboring frames, and just learn to auto-encode via the corrector). This is why we decode from the predicted state, which forces temporal prediction. For readout of down-stream tasks, we additionally make use of the corrected state, which combines both the prediction and the current observation.

“I think the prior work could go farther back in history in order to frame this proposed prediction-correction framework as being related to old school recursive filters. For example, I think if I were told how this is designed similar to a Kalman filter, then I'd have a much clearer picture of what is trying to be done.”

Thank you for pointing out the connection to Kalman filters and recursive filters. Indeed, there are some resemblances worth pointing out that we could comment on in the related work section. We will revise the text accordingly, which hopefully improves the overall presentation.

“I don't think Fig 1 does enough to clarify the system.”

Thank you for pointing this out. We intend for the revised draft to make the connection between the corrector and the predictor clearer. Further, we have updated the main model figure with this in mind, which can be found in the supplementary pdf attached to the general response. Please let us know if you have any suggestions for further changes.

“I didn't get a good impression of what kind of motion dynamics the predictor properly supported. It looked like there was some overshooting happening in the gifs, but it's not clear to me if this is a real issue. Can this be clarified?”

What the predictor would learn exactly is difficult to quantify as it will depend on many factors - model capacity, training data and so on. We do observe that when there is fast motion the model tends to lose tracking and assigns a new token to the moving element. This makes sense considering that the amount of uncertainty grows with faster motion. Whether this is a real issue will similarly depend on the nature of the down-stream tasks. Evidently, for our current evaluation spanning several standard computer vision tasks, this is not a major issue. However, for more complicated dynamics the predictor could benefit from a more specialized architecture.

“One ablation that would be useful is to change the backprop through time length. It would be great if there was evidence that only 2 or 3 frames were enough to get the predictor to estimate dynamics. Another question could be, what kind of dynamics are modeled with 2, 3, 4, 8, etc. frames and are there diminishing returns? Can this be clarified?”

Thank you for this suggestion. Generally speaking, we’d anticipate that a minimum of 3 frames is needed to make good predictions as acceleration information requires 3 different measurements - with 2 frames only velocity information is available. We agree that it would be interesting to ablate this hyper-parameter and we will commit to doing so in the updated version of the paper.

Previously we have explored training with a probabilistic “stop-gradient” that activated at random times through the sequence to effectively train on 3-4 frames long sequences but not overfit to a specific length. However as the model development continued, we no longer found this to be needed.

“Given that this model is deterministic and subject to over-blurring in the reconstruction task, have the authors explored other losses such as an LPIPS loss which might be more invariant to small misalignments?”

Thank you for this suggestion. In fact, we have not explored LPIPS or other perceptual losses to mitigate this, though this would be an interesting direction for future work. Integration of perceptual losses into MooG is not straightforward as we decode a random subset of pixels at every time step for efficiency reasons, as pixels directly serve as queries in the transformer decoder. Future work could explore decoding (latent) patches instead of raw pixels to mitigate this issue, which would make the model compatible with perceptual losses. A related direction that is worth pursuing is to have the state be the outcome of a sampling process (eg. as in a diffusion model or VAE), which might also help address blurry predictions.

Thank you for your detailed review and constructive comments.

“how to prove the effectiveness of the Corrector module?”

Thank you for your comment. Note that the corrector is the only part of the model that has access to the observation at the current time-step, i.e. to “correct” the prediction, and hence is the only channel for integrating new information. However, it also introduces a shortcut that bypasses the previous state when using the corrected state for reconstructing the observation. This is what necessitates both a predicted state (to be used for reconstruction that requires temporal prediction) and a corrected state (to be used for readouts that incorporate the current observation and is maximally informative).

“What’s the difference between MooG and Deformable DETR? Does the Corrector play a same role as the offset modal in Deformable DETR? Besides, do you treat the states as the tokens?”

One of the key differences between MooG and Deformable DETR is that Deformable DETR utilizes sparse spatial sampling of deformable convolution on the 2D feature map (with explicit multi-scale features), while MooG models the image as an off-the-grid set of latent representations. There is no sparsity nor offset in the corrector in MooG, whose purpose is to “corrrect” the state predicted from the previous timestep based on the current observation. The corrector cross-attends to the feature maps and updates the set of latent features through transformer layers directly. MooG is easier to implement, it is mainly attention blocks, there is no need for any specialized kernel (while deformable DETR requires specialized deform_attn_cuda.cu).

“It seems that you used transformer (attention module) in Corrector and Predictor, and how does the algorithm perform in tracking tasks? What is its speed? Can it meet real-time requirements?”

MooG is a very lightweight model, totalling approximately only 35M parameters during training (and fewer during inference). In particular, the transformers layers you mentionhave only 2 or 3 layers (predictor and corrector), while the conv-net is only 6 layers which should be no problem to run at real time on modern hardware.

Further, for readouts (like point/box tracking) the pixel decoder is not needed at inference time. The model performs well even when reducing the number of available tokens considerably (as shown in Figure 6 in Appendix C), which further improves speed.

“In table 3, the performance of TAPIR on Davis-full is better than MooG, so could you please provide more detailed explanations?”

Thank you for pointing this out. Indeed, because of the auto-regressive nature of the readout module, error accumulates when unrolling readouts over long sequences. This is not an issue with the base representation (as the corrector receives continuous observations) but we have observed how the read-out module ends up drifting eventually, i.e. as the initial conditioning signal (box or point tracks in the first frame) becomes less informative.

We note how TAPVid and TAPIR are domain-specific SOTA approaches that leverage specialized architectures, such as explicit cost volumes, to mitigate such issues for a particular domain. They further have access to both past as well as future states, while MooG is a causal model with access only to past frames, i.e. it can be used for online tracking. MooG learns representations that are useful for a variety of downstream tasks, and we have not incorporated domain-specific improvements in the readout decoder. One interesting direction for future work might be how to make MooG representations perform competitively on one single domain by incorporating more specialized components in the readout decoder. We will update the draft to give more context to this comparison and why we do not necessarily expect to beat domain-specific SOTA methods with MooG.

Thank you for your detailed review and constructive comments. We are pleased to find that you consider MooG to be an interesting and novel idea for self-supervised representation learning from videos, and that the presentation was clear to you.

“Please compare a) on generic tasks (evaluate learned representations for classification, object detection, or segmentation)”

Thank you for suggesting this comparison. MooG was designed with spatio-temporal grounding in mind, without placing a focus on high-level semantic features, and we chose evaluation domains that would demonstrate this most clearly. Keeping this mind in, we chose point tracking, box tracking, and depth estimation as low-level tasks that require little semantic understanding of a scene, which aligns well with our goal of learning scene-grounded, temporally-consistent representations. In preliminary experiments we observed that features learned by MooG are not well-suited for semantic tasks such as action classification in combination with linear readout heads. This is potentially due to the local nature of the representation or the simplicity of the readout setup we tried. We are happy to include a discussion of these preliminary observations in the paper and make suggestions for improvements for future follow-up work.

“It is unclear how these representations compare against video SSL methods [2,3,4,5]. Maybe try to apply some of these methods on the selected tasks to compare how MooG representations compare to them?‘)”.

Thank you for pointing this out. We have now added a comparison to VideoMAE v2 in the supplementary pdf attached to the main response, which is a more recent version of [5].

We consider representations from 3 publicly available VideoMAE v2 checkpoints: the ViT-small and ViT-base variants, which contain 22M and 83M parameters respectively, as well as a ViT-giant model (1B params). The smaller variants were obtained by distilling the predictions of the ViT-giant model. We note that MooG contains approximately 35M parameters, which includes the pixel decoder.

There are other differences that further make it difficult to compare, for example MooG was pre-trained on MOVI-E, while the VideoMAE models were trained on a superset of all Kinetics videos. MooG was trained purely in a self-supervised manner, while the VideoMAE v2 teacher network was finetuned for action recognition, and the ViT-small and ViT-base models were initialized from finetuned networks too.

Keeping these differences in mind, we observe how MooG performs significantly better than the ViT-small and ViT-base sized models that offer similar parameter counts. Compared to the teacher ViT-giant model that has 30x more parameters, MooG performs very well: it is better on MOVi points and Waymo boxes, but worse on MOVi depth and DAVIS points. Both methods perform the same on MOVi boxes. We are hopeful that scaling MooG to 1B+ parameters and larger scale pretraining may lead to further improvements, though we leave this for future work as mentioned in our limitations section.

“Related Works: Consider discussing [1] which explores a similar idea of next frame prediction in a latent space to learn good representations. Also [6] which learns local features using language which shows DINO like grouping / tracking behaviour.”

Thank you for pointing this out. We will make sure to contextualize our work further w.r.t. autoregressive next-frame prediction methods (e.g. regarding the reference you provided which uses a grid-based VQ-GAN representation [1]), and with perceptual grouping / object-centric methods in our related work section. In short: end-to-end object-centric grouping methods are indeed closely related (as alluded to in our related work section), but this line of work typically aims at grouping entire objects or well-defined parts into a single object – this includes methods such as GroupViT [7] and the paper you referred to [6]. For more general, fine-grained spatio-temporal tasks such as point tracking or monocular depth estimation, enforcing an object bottleneck (e.g. 1 token per object) may be too restrictive: we find that downstream task performance significantly increases by providing the model with more capacity in terms of number of “off-the-grid” vectors/tokens that it can use to encode the video.

References

[1] Bai et al., Sequential Modeling Enables Scalable Learning for Large Vision Models (CVPR 2024)

[2] Salehi et al., Time Does Tell: Self-Supervised Time-Tuning of Dense Image Representations (ICCV 2023)

[3] Venkataramanan et al., Is ImageNet worth 1 video? Learning strong image encoders from 1 long unlabelled video (ICLR 2024)

[4] Ranasinghe et al., Self-supervised Video Transformer (CVPR 2022)

[5] Tong et al., VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training (NeurIPS 2022)

[6] Ranasinghe et al., Perceptual Grouping in Contrastive Vision-Language Models (ICCV 2023)

[7] Xu et al., GroupViT: Semantic Segmentation Emerges from Text Supervision (CVPR 2022)

Thank you for your detailed review and constructive comments. We are pleased to hear that you found MooG interesting and novel, and recognized its considerable improvement over baselines in the frozen setting. We are glad that you found the writing and visualizations of high quality.

“One baseline that would be a closest comparison would be to train a Perceiver IO like model, except with spatio-temporal queries”

Thank you for suggesting this baseline, which is indeed a very sensible one. In fact it is close to one variant of the model we experimented with in the early phases of the project. At that time, we experimented with a non-autoregressive version of our model that mapped all encoded frames in parallel to a joint set of latent tokens (a single set for multiple time steps) and finally replicated this set across time steps with added temporal position encoding for decoding.

Such an approach is in fact very similar to Perceiver IO for video (with some small design differences in encoder, decoder and position encoding). In preliminary experiments, we found that when evaluated on 8-frame DAVIS point tracking, both models performed roughly equally well. However, we faced challenges continuing the point tracking predictions of the parallel (Perceiver IO-style) model when moving beyond the 8-frame window that it was trained on. On the contrary, this was straightforward to achieve with MooG due to its autoregressive formulation.

“Papers that use similar architecture but specialized for domain-specific video prediction tasks are missing from the related works”

Thank you for pointing this out. We will deepen the discussion of prior works in the related work section with regard to domain-specific approaches (such as SAVi), and highlight similarities and differences to MooG where relevant.

“Is the paper the first to show token binding to task related activities emerging unsupervised in a field-based transformer?”

To the best of our knowledge, this paper is among the first to explicitly study this relationship in standard transformer architectures trained in a self-supervised fashion. That said, it is difficult to make a precise claim about this due to numerous (domain-specific) prior works such as SAVi and PARTS exploring architectures that display similar behavior at a coarser scale (i.e. object-level binding) and are at least “transformer-like”. It is also worth mentioning the recent body of work on 4D Gaussian splatting (e.g. Luiten et al., Dynamic 3D Gaussians, 2023), which aims to learn explicit 3D Gaussian representations that track elements of a dynamic scene, typically by using more explicit geometric constraints instead of generic transformer-based architectures.

“Are the baselines chosen for point tracking (TAP-Net, TAPIR) and for depth prediction (DPT, etc) the state of the art results in the domains? […] a discussion on limitation and deficit to state of art (if any) would be useful).”

TAPIR is very close to state of the art in the point tracking domain, and only outperformed by the very recent BootsTAP [1], which extends TAPIR by augmenting the pretraining dataset with unlabeled YouTube videos. The DPT architecture itself is commonly used in competitive monocular depth-estimation approaches such as the MiDaS line of work [2]. However, to achieve competitive performance, such approaches are usually pre-trained on multiple different depth estimation datasets and adopt more sophisticated losses to handle depth targets from varying sources in a single model. More recently, diffusion-based approaches have shown great promise for monocular depth estimation [3, 4], which move away from the DPT architecture.

We agree that it would be useful to contextualize these baselines a bit better relative to SOTA in the field, and we will update the paper accordingly.

[1] BootsTAP: Bootstrapped Training for Tracking-Any-Point [2] MiDaS v3.1 -- A Model Zoo for Robust Monocular Relative Depth Estimation [3] Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation [4] Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data