Near, far: Patch-ordering enhances vision foundation models' scene understanding

Table 4. Linear segmentation (mIoU)

Table 5. Pascal VOC - Vision In Context Learning with NN Dense Retrieval (mIoU)

Table 6. ADE20k - Vision In Context Learning with NN Dense Retrieval (mIoU)

As shown in the tables above, NeCo improves the performance of both CLIP and SigLIP by approximately 12% to 37% across various benchmarks. These results demonstrate that NeCo is not limited to vision foundation models but can also be effectively applied to vision-language models.

Can it be trained from scratch, assuming that the similarities are given from another model?

Thank you for the interesting suggestion. We think this could be explored in future research, as this type of model distillation would be different from the main focus of our work. Our approach begins with a pretrained teacher network and improves its performance through further fine-tuning. In contrast, the model trained with the reviewer’s proposed method would likely be limited by the teacher/distilling network.

Would incorporating a repel signal similar to contrastive loss be helpful for the training?

The goal of this paper is to develop a feature space in which, for a given input, patches representing the same object exhibit similar features, while patches from different objects show distinct features. To effectively model this, we need a hierarchy of similarities. Contrastive methods, however, potentially violate this hierarchy by only aiming to bring similar patches together while pushing dissimilar ones apart, without explicitly enforcing the nuanced ordering that we require. In contrast, our sorting-based approach maintains this hierarchy by explicitly enforcing the relative similarity of patches within a more structured ordering. Moreover, sorting also generates repelling signals, as it brings similar patches together while simultaneously distancing them from dissimilar patches; yet, the repelling effect in sorting is milder than in contrastive methods.

We thank the reviewer once again for the valuable feedback. We hope the response adequately addresses the concerns raised and encourages the reviewer to improve the score.

More instance and dense-level tasks

More Dense tasks:

We appreciate the reviewer’s concern and have added linear segmentation performance on CityScapes and NYUd dataset to our linear segmentation experiments, as shown by Table. 2. Here are the results :

Table 1. Linear segmentation on Cityscapes (mIoU)

---

Table 2. Linear depth prediction on NYUd (Root-mean-squared-error, lower is better)

Backbone (ViT-S)

Backbone (ViT-B)

As demonstrated in the results, the features learned by NeCo consistently improve DINOv2R on different dense tasks and datasets, which support the generality of our method.

For the KITTI dataset, we expect results similar to those observed on Cityscapes. However, due to the large image sizes in KITTI, experiments take longer, and we plan to include the results in the final version.

Instance-level tasks:

The table below shows the performance of DINO and DINO fine-tuned with NeCo for object detection on the COCO dataset, evaluated using the VitDet benchmark with a ViT-small model. Our method improves the DINO backbone, increasing the validation Box AP slightly from 42.9 to 43.0 and the Mask AP from 38.6 to 38.7.

Table 2. Object detection performance based on VitDet benchmark

Could single-batch distance computation miss some semantically close patches?

We agree with the reviewer that single-batch distance computation may not capture all pairwise relationships between patches. We have done an ablation study on the impact of batch size on performance, as presented in Table 6.e. For clarity, we include the table here for reference.

Paper table 6. e: Ablation on batchsize

As shown, performance improves over the baseline even with smaller batch sizes, indicating that the most useful patches are likely already present in the image itself and not in the reference patches. However, we do observe gains with increased batch sizes, likely due to the increased presence of further semantically close patches.

How does the method handle scenarios where the foundation model produces noisy features?

As noted by the reviewer, the quality of the initial features allows our method to start with a strong pairwise distance matrix, which is further refined during training. We conducted an extensive ablation study on the effect of pretraining, as shown in Table 3. This includes results from five additional pretraining methods beyond DINOv2R, demonstrating that even less robust models serve as suitable starting points for our method.

Ablate on different similarity measures other than cosine

As requested by the reviewer we have added an ablation study on the distance metric used to create similarity metrics. As the table shows, Cosine similarity is consistently better than Euclidean distance. We have used cosine similarity in all our experiments.

Table 3. Linear segmentation (LS) and vision in-context performance (IC) on top of the frozen backbone

Add clip, sigclip or sam to your tables

We present the results for CLIP and SigLIP on linear segmentation and visual in-context learning, as requested by the reviewer.

--- continued ---

Dear Reviewer KF4P,

Thank you again for the time and effort spent on your thorough review of our paper. Since the author-reviewer discussion deadline is fast approaching, we kindly ask for feedback on our responses. We would be happy to discuss more if there are still some open questions.

Best Regards,

Authors

Increasing the batch size requires simultaneous adjustment of the learning rate:

We agree with the reviewer. In our work, we employed the ADAM optimizer, which dynamically adjusts the learning rate based on training dynamics. As a result, we expect that the adjustments in learning rate for different batch sizes would lead to a similar overall impact on the training process.

Additional experiment (i.e., batchsize = 1) should be conducted to verify the claim that "useful patches are likely already present in the image itself.":

As requested by the reviewer, we conducted an ablation study with a batch size of 1. Training with this configuration takes longer because processing each sample individually reduces the efficiency of computations. Consequently, the model was trained for only one epoch for this response. The results are presented in the table below:

Table 1. Evaluating the performance gain for batchsize=1

As shown, even with a batch size of 1 and just one epoch of training, NeCo demonstrates improvements in DINOv2R performance ranging from 0.4% to 7% across different metrics and datasets. Compared to our best results, approximately 30% of the linear evaluation gains and 80% of the in-context gains on Pascal VOC can be achieved using a single image in each batch. Similarly, for ADE20K, about 30% of the gains can also be achieved using patch comparisons within the image alone. Moreover, training with a batch size of 1 actually allows one to have a very small memory footprint of 3517MB during training, allowing us to actually improve the DINOv2R model with a single GTX 1080, a GPU from 2016. We find this surprising and interesting; thanks for the suggestion!

For further insights into the effect of patch selection, we had already included an additional ablation study in Table 10A (Appendix) and lines 505–509, which examines the impact of inter-image selection (ie image patches are compared to both patched within the same image and across patches from the references in the batch) and intra-image patch selection (ie only within the image) on performance. We add this table for your convenience below :

Table 10.e Nearest neighbor selection

As shown, the results for both are very close, with most of the gains achievable through intra-image patch selection, further supporting our claims.

NeCo's primary contribution is introducing sorting consistency in visual representation pretraining, though much of the Method section on differentiable swapping and sorting relies on prior work.

The reviewer is correct. This is to make the paper more self-contained and more straightforward for readers unfamiliar with those other works. We aim to make the prior work section even more compact.

Is the improvement coming from starting from a pre-trained network? Can the method be trained from scratch?

Most likely, yes, but training Vision Transformers (ViTs) from scratch requires many optimization tricks [1] which also significantly increases training time. Our main contribution lies in demonstrating that, with as little as 0.2% additional compute compared to DINOv2R, we can achieve substantial improvements in dense representation quality. This highlights the efficiency and practicality of our approach in contrast to the resource-intensive process of training ViTs from scratch.

improvement may partially come from the additional information brought by the pretrained backbone DINO.

Indeed, this is why we compare against the original pretrained models in all our experiments. Moreover, we also find that the gain does not solely come from using the COCO dataset either: From our experiment of training NeCo on PascalVOC, as shown in Table 6.d, we observe that the resulting performance—77.9 and 36.4 for linear segmentation on Pascal and ADE20K, respectively—exceeds that of the original model, which achieved 74.2 and 35.0, despite DINOv2's training dataset being constructed with PascalVOC. This improvement demonstrates that the proposed loss function imposes meaningful constraints on the dense features, enabling the model to better utilize the training samples, despite their prior exposure to the DINOv2R loss.

More details on sampling and collecting patch features with ROIAlign. How are different views sampled?(Question1)

Generating the Crops : We generate multiple views of an image by applying random cropping and resizing using the RandomResizedCrop method from PyTorch's torchvision.transforms module. For global crops, we use a narrower range of scaling parameters s=(0.25, 1), ensuring that larger regions of the image are captured. These global crops are resized to 224×224 resolution. For local crops, which focus on smaller regions, we use scaling parameters s=(0.05,0.25) and resize these crops to 96×96 resolution. Both types of crops are further augmented using horizontal flips, color distortions, and Gaussian blur to increase diversity. We follow this multi-crop strategy from previous works such as [3, 4].

Feature Extraction with ROI Align : For each pair of crops, we compute and store the intersection bounding boxes of their overlapping regions during the cropping process. During feature extraction, these bounding boxes are used with the ROI Align method to sample the corresponding regions from the feature maps. Since features from different crops may vary in size due to differing scales and resolutions, ROI Align resizes these regions to a fixed size (e.g., 7×7) using bilinear interpolation, ensuring spatial alignment and uniform dimensions.

We will make sure to add this explanation to our appendix.

How the sorting algorithm is implemented? Can we use nlog(n) algorithms?(Question2)

We have tested different sorting algorithms, as shown in Table 10.b (Appendix). Our method works well with all the algorithms, but bitonic sorting is slightly faster and more efficient than odd-even sorting, so we chose it as the default. Although we haven't tested our method with faster differentiable sorting algorithms, such as [2] with O(nlog⁡n) complexity, the modular design makes it easy to switch to a faster algorithm if needed, as long as it doesn't negatively affect performance.

We thank the reviewer once again for the valuable feedback. We hope the response adequately addresses the concerns raised and encourages the reviewer to improve the score.

[1] - Touvron, et al. "Deit iii: Revenge of the ViT." ECCV 2022.

[2] - Blondel et al. "Fast differentiable sorting and ranking." ICML 2020

[3] - Caron et al. "Emerging properties in self-supervised vision transformers." ICCV. 2021.

[4] - Ziegler et al. "Self-supervised learning of object parts for semantic segmentation." CVPR 2022.

Dear Reviewer 1JH6,

Thank you again for the time and effort spent on your thorough review of our paper. Since the author-reviewer discussion deadline is fast approaching, we kindly ask for feedback on our responses. We would be happy to discuss more if there are still some open questions.

Best Regards,

Authors

Dear Reviewer 1JH6,

we have aimed to thoroughly address your comments. If you have any further concerns, we would be most grateful if you could bring them to our attention, and we would be pleased to discuss them.

Sincerely,

Authors

Applying cross-entropy on the similarity matrix instead of the sorted ones:

Thank you for highlighting this interesting potential ablation. We have now conducted an additional experiment where we remove the sorting algorithm, and simply apply the cross-entropy loss on the similarity matrices, as requested by the reviewer. The results are presented below:

Table 1. Ablating the effect of sorting component. LS=Linear segmentation, IC=Vision in-context learning

The table is in the same format as our ablation studies (LS=Linear layer segmentation, IC=In-context segmentation). We find that removing the sorting component significantly reduces performance – though it does not lead to collapse. This can be attributed to the inability of a simple cross-entropy loss on similarities to provide sufficient discriminative power to separate dissimilar samples. Consequently, if the model is trained for enough epochs, this may lead to solutions suffering from mode collapse, where all features increasingly become identical. While the EMA teacher helps mitigate this issue to some extent, our experiments indicate that it is insufficient on its own. Incorporating a sorting component that explicitly orders neighbors with discrete rankings leads to a more effective and stable solution.

Comparing the generalization on out-of-distribution datasets such as Cityspaces (Question 1)

As requested by the reviewer, we report the performance of various backbones, including DINOv2R fine-tuned with the NeCo loss, on out-of-distribution datasets such as Cityscapes (semantic segmentation) and NYUd (monocular depth estimation). The evaluation follows the protocol outlined in Table 2 of the paper.

Table 2. Linear segmentation on Cityscapes (mIoU)

---

Table 3. Linear depth prediction on NYUd (Root-mean-squared-error, lower is better)

Backbone (ViT-S)

Backbone (ViT-B)

Table 4. Linear segmentation on Vaihingen - Remote Sensing (mIoU)

Backbone (ViT-B)

As demonstrated in the results, NeCo consistently enhances the generalization of all backbones on the Cityscapes dataset. Furthermore, even for tasks that diverge from semantic segmentation, such as depth estimation, NeCo reduces the DINOv2R error by around 2%. These findings highlight that NeCo not only maintains but improves the generalization capability of DINOv2R features across diverse tasks. For completeness, we also include results from [1], showing that NeCo consistently enhances the performance, even on out-of-distribution datasets.

Why are the full fine-tuning results not significantly better than the original DINOv2R (Question 2)?

Actually, in 3 out of 4 full-fine-tuning experiments, including the COCO-Stuff dataset for the base variant, our model outperforms the base DINOv2R, as shown in Table 4. However, for the small variant on COCO-Stuff, indeed our performance is slightly lower. This might be attributed to the high potential of full fine-tuning to adjust all the model’s weights. The original DINOv2R checkpoint benefits from its pretraining on LVM142M with 142 million images, providing a significant advantage by capturing a more diverse range of visual patterns and features. In contrast, the NeCo training dataset contains only 118K images—just 0.11% of the DINOv2R training dataset—which explains the slightly lower performance (0.3%) in some experiments. We are sure that applying the NeCo loss on a varied dataset such as LVM142M would bring even larger benefits.

We thank the reviewer once again for the valuable feedback. We hope the response adequately addresses the concerns raised and encourages the reviewer to improve the score.

[1] - Markus Gerke, I. T. C. "Use of the stair vision library within the ISPRS 2D semantic labeling benchmark (Vaihingen)." Use of the stair vision library within the isprs 2d semantic labeling benchmark (vaihingen) (2014).

Dear Reviewer 7M3r,

Thank you again for the time and effort spent on your thorough review of our paper. Since the author-reviewer discussion deadline is fast approaching, we kindly ask for feedback on our responses. We would be happy to discuss more if there are still some open questions.

Best Regards,

Authors

An ablation on matching training flops/epochs between all the pretraining methods and NeCo, including DINOV2R:

Although we fine-tune DINOv2R for 25 additional epochs, it is important to note that this model, which serves as the main model for NeCo, undergoes these epochs on COCO—a dataset that constitutes only 0.11% of DINOv2R's original training dataset of 142M images. In effect, the 25 training epochs equate to roughly 2% of a full epoch and thus merely 0.2% of DINOv2R's extensive pretraining (10 epoch pretraining). Despite this minimal training, NeCo improves DINOv2R's performance by at least 6% across several evaluation benchmarks. We also acknowledge the reviewer’s concern and have provided a detailed computational analysis in Table 11 (Appendix). Following previous methods [4], we report the total training and epoch time. We will move this table from the appendix to the main paper for visibility. The table has been extended with the results below, and epoch time is adjusted based on dataset size.

Table 1. Computational efficiency analysis and evaluation on frozen clustering with K=ground truth or 500 and linear segmentation

The results, reported on COCO-Things linear segmentation, highlight NeCo's significant improvement in computational efficiency and performance. First, DINO and CrIBo are finetuned for 25 additional epochs starting from their existing checkpoints to match the extra training performed by NeCo. As the table shows, with the same number of extra epochs, NeCo outperforms both models across all metrics, demonstrating that the improvement stems from the proposed loss function rather than extended training. Secondly, with only 2.5 GPU hours of extra training on top of CrIBo, NeCo boosts its performance in linear segmentation by 3.7%. These results show that NeCo not only enhances computational efficiency but also achieves superior results.

The method has carefully crafted hyperparameters to ensure some overlap between teacher and student patches

The augmentation procedure in our work adheres to the standard multi-crop strategy, as established in prior works such as SwAV, DINO, Leopart [1, 2, 3]. This approach generates both global and local views of the image: global crops capture broader context by covering larger portions of the image, while local crops focus on smaller regions to extract fine-grained details. As our loss is dense, we thus require some matching between the corresponding patches. For this, we simply follow [3], to ensure a positive intersection between global crops and align the views by ROIAlign.

Is it possible to remove the ROIAlign?

Yes, If multi-crop augmentation is omitted, ROI Align can also be removed by using the same Global Crop across the teacher and student branches. However, this removal could negatively affect performance due to using weaker self-supervised supervision through the augmentations. We have run this as an interesting ablation study and the results are reported in the table below:

Table 2. Ablating the effect of ROI Align operator

We thank the reviewer once again for the valuable feedback. We hope the response adequately addresses the concerns raised and encourages the reviewer to improve the score.

[1] - Caron et al. "Unsupervised learning of visual features by contrasting cluster assignments." NeurIPS 2020

[2] - Caron et al. "Emerging properties in self-supervised vision transformers." ICCV. 2021.

[3] - Ziegler et al. "Self-supervised learning of object parts for semantic segmentation." CVPR 2022.

[4] - Lebailly et al. "CrIBo: Self-Supervised Learning via Cross-Image Object-Level Bootstrapping." ICLR 2024.

Dear Reviewer hnaH,

Thank you again for the time and effort spent on your thorough review of our paper. Since the author-reviewer discussion deadline is fast approaching, we kindly ask for feedback on our responses. We would be happy to discuss more if there are still some open questions.

Best Regards,

Authors

Dear Reviewer hnaH,

we have aimed to thoroughly address your comments. If you have any further concerns, we would be most grateful if you could bring them to our attention, and we would be pleased to discuss them.

Sincerely,

Authors