VESSA: Video-based objEct-centric Self-Supervised Adaptation for Visual Foundation Models

We sincerely thank the reviewer for the constructive and detailed feedback. We are glad that the reviewer found the paper well written, with clear figures, carefully designed modules, and thorough evaluations. Below, we address each of the concerns.

W1: [Inadequate experiments] More experimental results are necessary for varifying the efficiency and effectiveness of the method. (i) Ablations on the number of local crops: In Section 3, the paper mentioned that a set of u pairs of local crops are processed by the student model. But it is unclear what this number is set and its influence on the final performance. (ii) Training cost comparisons on video-based and image-based methods. Although VESSA outperforms its static counterpart, this method requires paired input as well as local crops during training. Comparisons on the training cost would be more intuitive to show the efficient adaptation progress.

(i) Ablations on the number of local crops

We appreciate the suggestion to further analyze the impact of the number of local crops. In our experiments, we adopted a fixed configuration following standard practices in state-of-the-art self-supervised methods (e.g., DINOv2), which has proven to be effective and stable across different datasets. Due to space constraints and in order to maintain focus on the primary contributions of our work, we did not include a variation of such parameters. However, we would like to remind the reviewer that we already include an experiment without local crops in Table 1 (row 2), which achieves an accuracy of 90.53, while the version using the default DINO number of local crops reaches 91.87. The improvement using local crops is significantly smaller than the improvements using video inputs (compare row 1 to row 8: 91.87 vs 88.54) or the improvements by adapting the training head before updating the backbone (compare row 1 to row 7: 91.87 vs 80.87). Therefore, we believe that variations in the number of local crops would not bring as significant gains as the other aspects we emphasize as main contributions in our work.

(ii) Training cost comparisons between video-based and image-based methods

We agree that reporting training costs is valuable to assess the efficiency of VESSA. Our method introduces a minimal computational overhead compared to its static counterpart. In the image-based method (VESSA variant that receives only images as input, Table 1 row 8), each training step processes two augmented views of the same image. In the video-based method (VESSA variant that receives video frames as input), each training step processes two distinct frames from the same video. In both cases, the model processes exactly two images per step, leading to nearly identical GPU memory usage and computation cost.

The primary additional cost comes from video decoding, but this step is performed offline as a preprocessing stage, where all videos are decoded and frames are extracted at a fixed FPS before training. During adaptation, the training pipeline only samples pre-extracted frames, resulting in negligible additional cost and no significant impact on the overall throughput of model optimization.

It is also important to note that VESSA is not trained from scratch. Instead, it builds upon the publicly available pre-trained weights of DINOv2.. For our unsupervised adaptation stage, we measured the actual training cost (details also provided in our response to Reviewer 3, jCuG). Using a ViT-Base backbone on the Co3D dataset (20,412 training pairs, 4,535 test samples), the adaptation stage required only 1.97 hours of training (7.04 core hours) on a TPU v3-8, which consists of 4 TPUs, each with 2 cores (total of 8 cores) , consuming 13.86 kWh of energy and resulting in a carbon footprint of just 5.55 kg CO2 for 20 epochs.

This is negligible compared to the original DINOv2 training cost: according to their paper, reproducing DINOv2 alone requires 22,016 GPU-hours (9.7 MWh), emitting approximately 3.7 tons of CO2. The entire DINOv2 project is estimated to have a footprint between 0.5k and 1k tons of CO2 (around 200k GPU-days). In comparison, our adaptation stage emits a lot less CO2 than retraining DINOv2 from scratch.

Given the substantial performance gains demonstrated in Tables 1 and 3, we believe this trade-off is favorable. We will include runtime metrics in the final version of the paper and provide scripts in the released code to facilitate further cost analysis by the community.

W2: [Need further explorations] Although the method has shown potential in classification task, it would be more convincing to show whether this framework is robust and generalizable to other tasks. (i) In Figure 3, based on the retrieval results, VESSA focus more on object, which may benefit from the object-centric videos during adaptation. It would be more convincing to show whether such representation improvement can generalize to other object-centric tasks and more quantitative results are encouraged. Instance-level recognition experiments as validated in DINOv2[1] are recommended (but not necessary). (ii) Since VESSA incorporates temporal dynamic information, comparisons on video classification tasks with foundation models are also encouraged.

Our primary goal in this work is to develop an effective unsupervised adaptation method tailored to specific downstream tasks, rather than creating a universal model that performs optimally on a wide variety of tasks out of the box. This task-specific adaptation focus is highly relevant in practical applications, where efficient tuning to the target domain is often more valuable than large-scale pretraining alone. While we currently do not include instance-level recognition or video classification experiments, we acknowledge the reviewer’s comment that such experiments are not strictly necessary. We agree that, with appropriate adjustments, VESSA’s use of temporal dynamics and object-centric video information could be extended to these tasks. Extending our evaluations to these settings is part of our planned future work. We view these extensions as promising research directions that naturally follow the contributions of this paper.

We greatly appreciate the reviewer's thoughtful and detailed feedback. We are glad that our experiments are recognized as comprehensive, validating our method’s general applicability and robustness. Additionally, we appreciate the recognition of the method’s potential for real-world applications. The suggestions will be incorporated into the final version of the manuscript. Please find our detailed responses and clarifications below.

Q1 (W1): Training Cost Clarification: Please provide detailed information about the training cost, including total training time, computational resource consumption (e.g., GPU/TPU hours), and differences across models or dataset scales, to better understand the method’s practical feasibility.

Thank you for raising this question. These details are indeed important and will be included in the final version of the paper. Below, we present an analysis from a single execution of VESSA in an adaptation process for the Co3D dataset (20,412 training pairs and 4,535 test samples) using the ViT-Base architecture. The adaptation required 1.97 hours of training, corresponding to 7.04 core-hours. The total energy consumption was approximately 13.86 kWh, which translates into a carbon footprint of 5.55 kg CO₂, assuming an average emission factor of 0.4 kg CO₂/kWh (typical for data centers). The process ran for 20 epochs, processing a total of 407,040 examples at a throughput of 135 images/second (or 16.88 images/second per core), with an average rate of 1.0485 steps/second. The experiment used 8 devices in parallel, each with a local batch size of 128 and a per-device batch size of 16. The ViT-Base model contained 111,472,896 trainable parameters (including Lora parameters) during adaptation. It is worth noting that, unlike DINO, which trains from scratch, VESSA adapts a pre-trained model. Even considering this distinction, when compared to the reported training cost of the original DINO model on ImageNet, as described in Emerging Properties in Self-Supervised Vision Transformers (Caron et al., 2021), the efficiency gains are clear. DINO’s full pretraining involved approximately 1,024 GPU-hours on V100 (equivalent to hundreds of kWh and a proportionally higher CO₂ footprint), while VESSA completes adaptation in less than two hours, with only a small fraction of the computational and environmental cost. This makes VESSA a computationally efficient and environmentally sustainable option for domain adaptation.

Q2 (W2): Data Dependency and Generalization: How does the method perform when access to dense multi-view object-centric videos is limited? Could it be adapted or generalized to settings with fewer or no multi-view frames?

We would like to emphasize that the primary goal of our method is adaptation using object-centric videos, for which it performs well. In line with the idea of expanding the applicability of VESSA, there are several potential directions for future work. One promising avenue would be to incorporate an object tracking stage prior to applying VESSA. This tracking component would be responsible for following and extracting multiple views of the same object, enabling a multi-view VESSA setup even in unconstrained video environments. Such an extension could broaden the applicability of the method to scenarios with fewer or no dense multi-view frames.

Q3 (W3): Forgetting Mitigation: Have you explored additional strategies to mitigate forgetting during fine-tuning beyond partial layer unfreezing and LoRA? Quantitative forgetting metrics or further regularization techniques would strengthen the analysis.

We acknowledge the concern regarding the potential impact of domain-specific adaptation on performance in broader, more general tasks. Our approach is deliberately designed to tailor the model to a specific target domain, where it consistently delivers strong results. Nonetheless, recognizing the relevance of this question, we conducted an additional evaluation to measure classification performance on the ImageNet dataset using a KNN classifier. In this assessment, we compared the visual foundation models under consideration (DINO, DINOv2, and TIPS) both in their original pre-trained state and after adaptation with VESSA on each of the target domains (CO3D and MVImageNet).

The first row of the table reports the baseline performance of each pre-trained visual foundation model when evaluated on ImageNet. The second row presents the results on the same dataset after adapting the models with VESSA using the CO3D data, where a substantial decrease in accuracy can be observed. A similar reduction appears in the third row, corresponding to models adapted on the MVImageNet dataset. These findings highlight that the adapted models are not suitable for general-purpose use. However, it is important to emphasize that, in practical scenarios, the adaptation is intended for a specific target domain, where performance continues to improve and remains highly effective for that domain.

W4: Some Implementation Details Could Be Clearer: Additional details on hyper-parameter choices would improve reproducibility.

We appreciate this observation and will include more comprehensive details on the hyperparameter choices in the final version of the manuscript, noting that all implementation details will also be made available in our GitHub repository. The VESSA learning technique was implemented on top of DINO, primarily to benefit from its stable training setup and the extensive empirical tuning described in the original DINO work, as well as the similar training environment it offers. Consequently, many update-related hyperparameters were inherited directly, including: global_crops_scale (0.14, 1.0), local_crops_scale (0.05, 0.25), student_temp (0.1), center_momentum (0.9), number_local_crops (10), warmup_teacher_temp (0.04), teacher_temp (0.07), learning rate (0.001), learning_rate_schedule ("compound"), factors ("constant * cosine_decay * linear_warmup"), warmup_steps (config.steps_per_epoch * 15), steps_per_cycle (total_steps), base_learning_rate (lr * batch_size / 1024.), and alpha (0.01). Some settings were modified to meet machine memory limitations and to incorporate VESSA-specific adjustments. These include: epochs (20), lora_rank (64), batch_size (128), gamma (UWSD) (1.0), and number_pairs_video (3). These modifications were chosen to align with the available computational resources while preserving the method’s effectiveness.

We trust that these answers offer a more comprehensive view of VESSA’s training cost, data requirements and generalization behavior, as well as its handling of forgetting and implementation aspects. All points raised will be incorporated into the final version of the paper.

We sincerely appreciate the reviewer's insightful feedback. Several strengths of our method were highlighted, including its motivation, clear/intuitive approach and its lightweight training. The experiments are regarded as comprehensive,validating the method’s effectiveness. We have carefully considered all the feedback and provided comprehensive responses below. All the suggested points and clarifications will be incorporated into the final version of the manuscript.

W1: It is unclear whether key implementation details from the original DINO framework, such as temperature setting and teacher centering, were utilized. Clarifying this and the specific hyperparameters would help disambiguate the performance gains attributable to the proposed methods (e.g. UWSD) versus standard training techniques

The implementation of the VESSA update component was based on DINO for training stability reasons, given the extensive empirical search reported in the DINO work and the similar environment it provides. In particular, the temperature setting and teacher centering were utilized, exactly following the DINO setup. To make it absolutely clear, we list the hyperparameters in detail here: global_crops_scale (0.14, 1.0), local_crops_scale (0.05, 0.25), student_temp (0.1), center_momentum (0.9), number_local_crops (u = 10), warmup_teacher_temp (0.04), teacher_temp (0.07), learning rate (0.001), learning_rate_schedule ('compound'), factors ('constant * cosine_decay * linear_warmup'), warmup_steps (config.steps_per_epoch * 15), steps_per_cycle (total_steps), base_learning_rate (lr * batch_size / 1024.), and alpha (0.01). Other key parameters were adapted due to machine memory constraints and hyperparameters specific to VESSA. These are the key adaptations: epochs (20), lora_rank (64), batch_size (128), gamma (UWSD) (1.0), and number_pairs_video (3). These changes were made to ensure compatibility with the available computational resources while maintaining the effectiveness of the method. All the details will be made available on GitHub for transparency and reproducibility.

W2: While the paper acknowledges catastrophic forgetting as a limitation, the extent to which it affects the model's performance on its original pretraining tasks is not explored.

We understand the concern regarding the impact of our domain-specific adaptation on more general tasks. We would like to emphasize that our goal is to adapt the model to a specific domain and apply it solely within that target domain, where it performs very effectively for this purpose. However, acknowledging the relevance of the question, we conducted an experiment to evaluate the classification results using KNN on the ImageNet dataset. In this experiment, we compared the performance of the visual foundation models in question (DINO, DINOv2, and TIPS) both pre-trained and after adaptation with VESSA on each of the domains (CO3D, MVImageNet).

The first row of the table presents the results for each pre-trained VFM validated on ImageNet. The second row shows the validation results on the same ImageNet dataset after adaptation with VESSA on the CO3D data. It is evident that there is a drastic drop in performance. The same drop is observed in the third row, where the models are adapted on the MVImageNet dataset. This demonstrates the infeasibility of using the model in a general manner. Nonetheless, we reiterate that our goal is to adapt the model to a specific target domain without supervision, achieving excellent results in this setup.

Q1: The experiments are conducted on datasets that feature largely object-centric videos. Could you elaborate on the expected performance and potential challenges of applying VESSA to more general, "in-the-wild" videos that may contain significant background clutter, camera motion, or multiple objects?

Thank you very much for the suggestion on ways to expand the application of VESSA. Some of these ideas are being considered for future work that will explore this direction. One potential solution to address the challenges posed by background clutter, dynamic camera motion, and multiple objects is the incorporation of an object tracking method prior to the application of VESSA. This approach, while relying on the quality of the tracking method to ensure effective object localization, would allow VESSA to extend its capabilities to handle scenes with multiple objects and significant camera motion. In this case, this expansion would enable VESSA to perform well even in more chaotic real-world video environments, albeit with some dependence on the precision of the object tracking method employed. We hope these responses provide a clearer understanding of the hyperparameters, the impact of adaptation on the visual foundation models, and a potential expansion in the application of VESSA. We greatly appreciate the feedback and look forward to further enhancing our work.

We sincerely thank the reviewer for the thoughtful and constructive feedback. We are glad that the reviewer found the paper well-motivated, sound, and interesting, with clear writing and thorough ablations. We appreciate the suggestions for improving clarity, refining the scope and framing of the paper, and exploring additional evaluations. Below, we address each of the reviewer’s concerns in detail.

W1: Limited applicability: While the method is being advertised as a video-based adaptation mechanism for VFMs, it’s only designed to work on multi-view object centric datasets, i.e. it is not applicable to any in-the-wild video. With this in mind, I think the framing of the overall paper, including the title, should be narrowed to this use case

We acknowledge that VESSA was specifically applied to object-centric videos in this work. We agree with the reviewer that clarifying this scope in the framing and title of the paper would improve its accuracy, and we are committed to making these changes. Despite this focus, the method already achieves significant improvements over the baseline on challenging multi-view datasets, demonstrating its practical value. We also recognize the importance of extending VESSA to handle in-the-wild videos with more complex dynamics, which we consider an important direction for future research. One potential avenue for this extension is the incorporation of an object tracking method prior to applying VESSA. Such a pre-processing step could help isolate objects in more cluttered scenes with multiple objects or significant camera motion. While we have not experimentally validated this yet, we believe this approach could allow VESSA to operate effectively in more realistic video scenarios, and we plan to explore it in future work.

W2: The main VFM pre-training is not significantly changed, the model is trained with the standard loss with an added Uncertainty Weighting. This makes the technical contribution a bit thin.

It is important to emphasize that our contribution goes far beyond simply adding the Uncertanty-Weighted Self-Distillation (UWSD) loss. VESSA introduces a comprehensive self-supervised adaptation framework for vision foundation models (VFMs), specifically designed to leverage video information efficiently. To ensure stable adaptation, we propose a staged unfreezing approach, where we (i) initially freeze the backbone and train only the projection head, (ii) progressively unfreeze layers, and (iii) apply LoRA-based low-rank adaptation to early transformer layers, while fully fine-tuning later layers. This carefully designed strategy mitigates gradient instabilities and preserves valuable pretrained features. These combined elements form a lightweight yet effective adaptation method that consistently improves classification accuracy on domain-specific datasets (up to +10.54% in Table 3) while requiring minimal computational overhead. We therefore respectfully disagree with the characterization of the contribution as “thin”; rather, we believe the paper offers a significant and practically relevant step toward unsupervised adaptation of VFMs.

W3: In Line 253, the authors cite their motivation for using temporal information to improve the VFM. While this is technically not false, I think describing at „using views of an object from a different angle“ would probably refine it.

We appreciate this observation. While we agree that the goal of our approach is to leverage different views of an object, it is important to clarify that the benefit comes specifically from using frames of the same physical object captured over time in a video, rather than simply using different objects of the same category viewed from different angles (e.g., a front view of one wooden chair and a side view of a different plastic chair). Using frames from the same video ensures that the model learns consistent, object-centric representations and focuses less on background information, as also highlighted by our qualitative results. To make this distinction clearer, we will refine the phrasing to “using different views of the object from the video” in the final version of the paper.

W4: Table 1 is overfilled and a bit confusing. It’s hard to piece together what actually works and how doesn’t different numbers of unfrozen layers and the other components. Also, using local crops seems to bring the biggest increase in performance, a technique already used for the original DINO. This begs the question how relevant the proposed technical contributions are for the performance in relation.

For the final version, we will expand the results and improve the presentation of the tables to make the contributions of each component clearer. We would like to clarify that, contrary to the impression that local crops bring the largest performance gain, other components — particularly training the classification head and using video as input — have a much stronger impact on the results. For example:

• Using images as input with local crops (Table 1, top section, last row) achieves 88.54% accuracy, while switching to video input without local crops (Table 1, top section, row 2) already improves performance to 90.53%, a much larger gain than local crops alone.
• When combining video input with local crops and training the head, accuracy reaches 91.87%, whereas using the same video setup with local crops but without training the head (Table 1, top section, penultimate row) drops to 80.87%. This demonstrates that training the head and leveraging temporal information from videos are the dominant factors driving VESSA’s improvements, while local crops provide additional but comparatively smaller gains. The ablation table was designed to explicitly highlight these decisions and show why each proposed step (video-based adaptation, staged unfreezing, head training, local crops, and UWSD loss) is necessary. We will make these findings clearer in the final version to avoid misinterpretation.

W5: Tables 1 and 2 are both ablation tables and probably should be moved to a separate ablation section.

We appreciate this suggestion. To improve the organization and clarity of the results presentation, we plan to create a dedicated ablation subsection in the final version of the paper. This subsection will group Tables 1 and 2 together and provide a more structured discussion of the ablation findings, making it easier for readers to follow the contribution of each component.

W6: It would be interesting to learn if the fine-tuning of DINO with multi-view datasets have improved its performance on benchmarks other than CO3D and MVImageNet

We would like to emphasize that our focus is not on creating a universal model that performs well across all multi-view datasets. Instead, VESSA is designed for efficient, domain-specific adaptation of visual foundation models. To address the reviewer’s curiosity, we conducted an additional experiment where we adapted the visual foundation models DINO and DINOv2 to MVImageNet and evaluated them on CO3D, aiming to understand the behavior when the model is adapted to one dataset and tested on another.

The results in the above table show a clear degradation in performance for DINO and DINOv2 in the cross-dataset scenario, indicating that adaptation is domain-specific and that improvements on one multi-view dataset do not automatically transfer to another. This behavior could be seen as a limitation if the goal were to adapt the model to multi-view datasets generally, however our motivation is to apply VESSA towards specific domains, as previously mentioned. This is motivated by real-world applications, where practitioners often need to adapt a model to a specific target dataset or domain efficiently, rather than aiming for a single model that generalizes equally well across all possible scenarios.

W7: Extending the approach to other VFM such as iBOT [1] or I-JEPA [2] would also be interesting

We appreciate the reviewer’s suggestion to evaluate other VFMs. We conducted additional experiments applying VESSA to iBOT on the CO3D dataset. using the publicly available ViT-base weights from the iBOT student network. (pre-trained on ImageNet-22K), the accuracy improved from 62.79% to 63.70% after adaptation with VESSA. Unfortunately, I-JEPA does not release ViT-Base weights, which prevented us from performing similar experiments for this model. We will include these results in the final version of the paper.