Annotation Bootstrapping: Reinforcing Visual Pre-Training using Unlabelled Images

Thank you for your thoughtful review! We have updated the paper to include comparisons to I-JEPA in Table 1 and 2; we find that at our data and model scales, annotation bootstrapping outperforms I-JEPA across our domains. We acknowledge that the current draft of our work has many shortcomings in presentation and clarity, and we greatly appreciate your detailed comments to help us improve the manuscript for the future.Please find responses to your specific questions below:

How well does this method perform when compare with I-JEPA

We have added a comparison to I-JEPA (using the open-sourced code) for all of the fully unlabelled experiments – the results have been added respectively to Tables 1, 2, and 3 in the updated revision. I-JEPA performs relatively poorly on all of these domains (underperforming SimCLR, DINO, and MAE). This is consistent with the original I-JEPA paper, which found performance to be sensitive to the choice of masking scheme (and consequently, the image distribution) and to smaller model sizes, with large degradations from ViT-H (their primary model) to ViT-B (smallest model they evaluated).

Does this method use a pre-training stage?

In our experiments, the models are trained from scratch, and all of the pre-training is done simultaneously on the labelled and unlabelled datasets. It would be interesting future work to understand whether this pre-training process can be initialized from a pre-trained model, e.g. a pre-trained CLIP / SigLIP checkpoint.

What is the structure of g? How can it accept both the representations and the transformations as inputs?

$g$ is a vanilla transformer decoder, identical to a cross-attentive VLM like Flamingo, albeit smaller. This decoder model takes in a sequence of tokens describing the transformation (akin to the “text” in the VLM); at each layer, these tokens cross-attend to the output of the ViT encoder. This allows the model to accept both the image representation (through cross-attention) and transformations (through the inputs).

The method is only evaluated under ViT-S. It will be more convincing to provide additional results with (at least) ViT-B.

In a future revision, we plan to add an experiment scaling the model size on the CC12M dataset -- we were unable to do so for the discussion period, since we are unfortunately limited by compute.

Thank you for your thoughtful review! We acknowledge that the current draft of our work has many shortcomings in presentation and clarity, and we greatly appreciate your detailed comments to help us improve the manuscript for the future. Please find detailed responses to your questions below:

L 464-471. If smaller IoU leads to better performance, why don't we train on non-overlapping crops?

We do! In our main experiments, we use the standard random resized crop RRC(8, 100) to sample crops (which w/ some probability, results in non-overlapping crops). The experiment in Figure 3 was a controlled experiment to determine the impact of the overlap between different crops

L.13-15: This is not correct. Many existing methods work learn well from web datasets or even satellite images (DINOv2, Tolan et al. [1], Vo et al. [2], AIM [3]).

We apologize for the over-statement. Our intention was simply that careful curation of these raw web-scale datasets is an important ingredient of many of these efforts, e.g. exemplar-based automatic data curation in DINOv2, and ImageNet mixture sampling in AIM.

From this point of view, the proposed framework is very similar to DINO or SwaV… (L_boostrapping without considering the transformation)

The functional form of our update indeed looks similar to DINO/SWAV, but the choice of “prediction space” leads to significantly different learning styles. In DINO, the prediction space is recursive, predicting the features that themselves were used to optimize the prediction objective. In contrast, for annotation bootstrapping, the prediction space is not recursive; the model is requested to predict the “contrastive features” (coming from L_annotation), not the “bootstrapping features” (coming from L_propagation). The former encourages the encoder to learn features that can be easily predicted, leading to features that are shared and inferrable across “local” crops of an image. The bootstrapping objective makes no such inclination towards learning more (or less) predictable features, but rather defers the choice of what features to predict to the annotation objective. We hope to make the distinctions between our method and DINO / SWAV more explicit in a future revision of the methods section.

How exactly COCO captions and bounding boxes are used as annotations in Table 3?

Bounding boxes and captions are tokenized similar to standard VLM training pipelines. Bounding boxes are tokenized into the following format “<y0=0.1> <x0=0.0> <y1=0.5> <x1=0.3> Object Name”; Caption texts are directly tokenized. In this experiment, both the unlabelled and labelled training is done using COCO.

What is the performance if learnt prototypes are used as in DINO instead of such arbitrary projection to the annotation space?

This is an interesting experiment to run, and one we will add to the paper! We note though that the projection to the annotation space is not arbitrary; in fact, this projection means that both the annotation loss and bootstrapping loss are making predictions about the same underlying distribution (the annotation distribution associated with an image), in principle offering a canonical way to balance the two losses. It is not obvious whether this will hold when using learned exemplars as in DINO.

Thank you for your thoughtful review. We have updated our submission to include 1) results for I-JEPA and 2) traditional probing setups, including linear probing and full finetuning on Imagenet (Table 5), . We acknowledge that the current draft of our work has many shortcomings in presentation and clarity, and we greatly appreciate your detailed comments to help us improve the manuscript for the future. Please find responses to specific questions in your review below:

There should be comparisons to JEPA style architectures like I-JEPA

We have added a comparison to I-JEPA (using the open-sourced code) for all of the fully unlabelled experiments – the results have been added respectively to the updated revision. I-JEPA performs relatively poorly on many of these domains (underperforming SimCLR, DINO, and MAE). This is consistent with the original I-JEPA paper, which found performance to be highly sensitive to the choice of masking scheme (and consequently, the image distribution) and to the model size, with large degradations from ViT-H (their primary model) to ViT-B (smallest model they evaluated).

Please show evaluations using more standardized setups

We have updated the submission with Table 5, which measures Imagenet linear probing accuracy and full fine-tuning accuracy on the models pre-trained on Imagenet. Annotation bootstrapping remains the strongest method under full finetuning, although DINO performs slightly better under a linear probe. We note that linear probing disproportionately downweights 1) pre-training methods that do not pool encoder outputs in pre-training like MAE and I-JEPA, and 2) methods that pre-train on datasets visually distinct from Imagenet (especially COCO and Epic Kitchens).

What does “ copy (only) the annotations from the labelled batch” mean in L281? While bootstrapping are we using both labelled and unlabeled images?

The bootstrapping objectives uses only the unlabelled images, but still needs a set of annotations to generate a target distribution (i.e. to predict $p(\ell | s, a)$, we need to plug in some $\ell$). Since the unlabelled and labelled pre-training happens concurrently, the easiest way to implement this is to “copy” the list of annotations from the labelled batch to use for the unlabelled update. These two lists do not need to be the same however: we can use a memory bank of previously seen annotations or we can use a fixed set of annotations for the bootstrapping update.

How is the video training performed? There are no details about the approach around EK.

We treat EpicKitchens as a dataset of individual frames, to match the setup with the other domains in the paper. We have updated the paper to include this detail.