FeatSharp: Your Vision Model Features, Sharper

Thank you for your thorough review.

Limited Novelty: The proposed method primarily builds upon FeatUp, incorporating well-established concepts such as tiling for handling high-resolution images and a simple learnable buffer for de-biasing. The integration of FeatUp, tiling, and attention layers is straightforward, and given these components, the performance improvement over FeatUp is unsurprising. The paper does not introduce fundamentally new techniques or discoveries.

We disagree that pulling together disparate concepts that are seen in other domains of computer vision itself lends to limited novelty. Tiling is the best example of this. While it is used as a method to work around the inflexibility of most ViTs within the context of VLMs, or implicitly when doing things like sliding-window segmentation, the major insight in our work is that the tiles provide fine-grained guidance to the upsampler. This type of guidance is not present in other upsampler works, as they either rely on the raw pixels (which lack semantic meaning), or they rely on the encoder hierarchy (which isn't applicable to ViTs). ReSFU [2], for example, uses raw pixel guidance. The other critical point about upsampling has to do with the fact that small features are irrecoverable from a combination of raw-pixel + low-res featurizer. Methods like FeatUp's implicit model (>1000 views), and FeatSharp with tiling, however, are able to recover this detail because they're observing the small features. We can see this effect in Figure 13, SigLIP section, left column. These single input visualizations show what happens when you only get a single input source (plus raw pixels). We otherwise use the same model, including the single transformer block. Only the tiles observe enough detail to separate the text lines. For the RADIO section, relying on the JBU stack from FeatUp entirely misses the street lamp, blurring it into the background, and bilinear is unable to recover the latticing. Further, as part of our rebuttal to oKYc we further our upsampler in an object detection setting, with additional comparisons with SAPA and ReSFU, and find that FeatSharp consistently does better. Particularly, we make the largest gains on small objects, directly providing evidence that tiles allow for the introduction of fine-grained detail. The set of techniques we chose to incorporate (multiview consistency, de-biasing, tiling, attention) are not ad-hoc, but form a cohesive novel solution to a problem.

Limited Generalization and Marginal Gains: The paper evaluates FeatSharp exclusively within the RADIO model as the teacher, without testing its applicability to other architectures. This raises concerns about its generalizability. Moreover, the reported improvement over FeatUp is only 0.39%, which is almost negligible, calling into question the practical significance of the proposed enhancements.

We believe that this characterization is not supported by the evidence. We chose the RADIO training harness because it is currently the state of the art for general perception foundation models, and because it has a training setting that could directly benefit from a feature upsampler, specifically, the low-res-teacher/hi-res-student part of their training protocol. However, it's also not the case that this setting is a 0.39% improvement over FeatUp. FeatUp damages RADIO training, resulting in a -0.13% change across the benchmark suite. So FeatSharp is +0.52% relative. FeatSharp is also 0.6% better than the RADIO-AMP reported results. While 0.6% may seem small, we stress that this is for a single model over an entire suite of 31 tasks. Further, the RADIO training setting was one of two benchmark evaluations performed in the main body, the other being ADE20k semantic segmentation with six different base featurizers, and in all cases, FeatSharp was shown to be superior. In the appendix, we additionally study upsampling DFN CLIP and RADIO on Probe3d and NYUDv2 benchmarks. For an additional point of clarification, section 4.4 isn’t using an existing RADIO model as a teacher, but rather, we’re using the training protocol, and we’re upsampling DFN CLIP and SigLIP. We further chose SigLIP2-SO400M-512 as an additional featurizer in the rebuttal object detection study to provide even more evidence of broad applicability.

Insufficient Baseline Comparisons: The work only compares against FeatUp, assuming it to be the sole relevant competitor for feature map upsampling. This narrow comparison overlooks other state-of-the-art methods, limiting the credibility of the results. A broader evaluation against multiple competitive approaches is necessary to justify the effectiveness of FeatSharp.

Thank you for this feedback. We have included SAPA and ReSFU in the object detection study in our rebuttal to oKYc upon your advice.

We thank you for your review.

I'm curious about the performance on fine-grained image image classification datasets, like CUB-200-2011

Thank you for your suggestion. Due to time and compute constraints, we were limited to running only one further study comparing upsamplers, and we selected COCO 2017 detection due to its ubiquity in the literature. We have those results in this rebuttal to reviewer oKYc. Given the results on COCO, it is plausible that similar effects could be observed with detailed datasets, particularly with small foreground categories, such as Caltech-UCSD Birds-200-2011, but are forced to leave the analysis to future work.

The paper has some presentation mistakes. One fire sign in Fig. 1 is out of the box, and the colors of the boxes are confusing. The citation in Line 249 is confusing. One captioned data in Fig. 5 is out of the box. The presentation can be further promoted.

Thank you, we will absolutely revise these issues in a prospective camera ready.

Thank you for your detailed review.

what does "2x upsampling" exactly mean?

We increase the width and height by 2x.

However, the quantitative results are not very convincing to me. For example, in Fig. 7, the neumerical results on semantic segmentation, why does 3x sampling generally underperform both 2x and 4x? Intuitively, if the upsample method is correct, the semantic segmentation task should continuously benefit from the high-definition feature map. However, the authors observed counterintuitive results but did not provide explanations or analyses.

We agree that we're missing the analysis for the 3x case. However, it's not necessarily true that increasing the resolution of the features should increase semantic segmentation benchmarks. Our goal is to produce higher-resolution features that are consistent with the featurizer itself, and that is occasionally at odds with doing well on a particular downstream task. For example, resolution is not the reason that PaliGemma is worse than DINOv2-L which is worse than RADIOv2.5-L, because they all encode the same number of tokens. Instead, in figure 7, what we observe is that FeatUp/FeatSharp are cleaning up the noisy models by favoring the view-consistent representations (compared to the baseline horizontal lines). We further observe that FeatSharp consistently does better than FeatUp on the task. An alternative way to look at this is by comparing "Baseline Inpt-1x" against "Baseline Inpt-2x". This gives us a sense of whether it's merely increasing the resolution that yields better segmentation. We see that DINOv2-L and RADIOv2.5-L improve when increasing resolution. However, the noisy models (DFN CLIP, PaliGemma, SigLIP) either see negligible change, or even get a bit worse.

We observe a similar effect in tables 11, 12, and 13 where the optimal choice of upsampler depends on the task and the featurizer itself. For DFN CLIP in the NAVI Correspondence task, it appears as though a simple bilateral upsample works the best, with entirely untuned weights, owing to both encoder noise, and also perhaps OOD alignment between the model and task. For table 13, we see that FeatSharp operates similarly to running RADIO at the resolution that matches the number of output tokens, with FeatSharp always being better than the low-res baseline (512px input).

Additionally, in most experiments in the appendix, I also found that FeatSharp does not consistently bring quantitative gains across various visual prediction tasks. These issues raise concerns about the significance and contributions of the methods in this paper: Merely looking good in visualizations does not prove the method's effectiveness. The authors need more numerical superiority in the results to demonstrate the specific benefits of the new model for visual tasks.

Thank you for this feedback, as it has motivated us to run another benchmark setting in order to provide further evidence of the efficacy of our method. We have integrated our method, as well as SAPA [3], and ReSFU [4], into Detectron2, and evaluated on COCO 2017. We used Edge [5], which allowed us to create a <featurizer>+<upsampler>+DINO [6] harness. We chose to use RADIOv2.5-L and SigLIP2-SO400M-512 [7] as featurizers, with the latter to further demonstrate versatility.

RADIOv2.5-L (512px input)

SigLIP2-SO400M-512 (512px input)

** SAPA failed in backprop with a cuda configuration error, due to the size of the feature map and channel count of the model.

While FeatSharp improves the metrics across the board, it makes its largest improvements on AP Small, followed by AP Medium. This makes intuitive sense as the tiling allows for the detection of small objects that otherwise are missed by any upsampler which doesn't have access to multiple views. FeatSharp provides a holistic method which not only keeps representations consistent as resolution increases, but also allows for the incorporation of new details which are missed by the low res encoding, which is an important advancement of the literature.

[3] SAPA

[4] ReSFU

[5] Edge

[6] DINO (not to be confused with DINOv2)

[7] SigLIP2

We thank you for your thoughtful review.

Multi-view consistency / Fidelity Experiments: This measures the mse distance between warped-upsampled features and the encoder’s low-resolution features. While this ensures consistent alignment, it doesn’t guarantee improved semantic detail. One could achieve high fidelity by over-smoothing the features so these results are generally unconvincing but do provide some additional insight to their remaining experiments.

We argue that over-smoothing is actually what's happening with either bilinear upsampling, or even functionally how the JBU-stack is operating. Figure 6 shows how FeatUp 4x (JBU-stack) leads to over-smoothed results, but with edge preservation when there are strong enough edges in RGB space. This smoothness is why we compare FeatSharp against bilinear and FeatUp on fidelity directly in Figure 5, and FeatSharp consistently does better. So it's not that smoothing in the upsampled space doesn't work, but rather that it doesn't work as well as what FeatSharp is doing. Fidelity is telling us how internally consistent the representations are with respect to a particular model, under arbitrary crops and deformations. Over-smoothing would cause issues with fidelity when the crops are small, since nearly all variation will be gone.

My main concern is the lack of comparison against FeatUp (with implicit upsampling). The exclusion of FeatUp's implicit model is understandable given its computational cost (~1 min/sample, Appendix G). However, a small empirical comparison—at least in terms of performance and training time trade-offs—would significantly strengthen the paper's claims. Can you better justify why this is not feasible to do, at least for smaller scale experiments and add it to the paper?

Upon your feedback, we have looked into this. It is ~1 min/sample for only the ViT-S/16 featurizer, but balloons quickly for the larger featurizers, like ~5 min/image for SigLIP2-SO400M-512, or ~4 min/image for RADIOv2.5-L/16 at 512px. However, the official implicit upsampler was also projecting the feature dimension down to 128 using PCA to achieve this speed [1, 2]. For ViT-S/16, it’s about 2x slower to use full features (e.g. 2 min/image). Due to the cost of running this mode, we ran a limited study on 100 images from COCO 2017 validation, and found that the fidelity is 3.17, higher than the 2.42 found with FeatSharp. However, FeatSharp at the same 16x upsample ratio takes 250ms/image (500x faster). These speeds are using a single H100 gpu.

Can you clarify why RADIO-AMP-L is performing worse than the baseline in Table 1?

We did our best to match the training setting between our reproduction and their claimed results. A key difference seems to come down to the fact that in RADIO-AMP, they bilinear downsample the student to match the teacher in the hi-res-student/low-res-teacher setting, whereas we bilinear upsample the teacher to match the student in our work. We made this change so that we'd have a comparison with a true upsampling baseline.

The proposed methods and evaluation are sensible, although only segmentation results on a single dataset (ADE20k) are shown for multiple of the vision encoders (DINOv2, SigLIP, SAM). More extensive evaluation is done to show the benefit on incorporating FeatSharp within the AM-Radio method (eg. Classification: imagenet1k - Zero Shot Retrieval: COCO + Flickr30k…) but overall the methods and evaluation are reasonable.

We have also included further benchmarking of our approach on object detection in response to reviewer oKYc below.

[1] https://github.com/mhamilton723/FeatUp/blob/main/featup/train_implicit_upsampler.py#L197

[2] https://github.com/mhamilton723/FeatUp/blob/main/featup/configs/implicit_upsampler.yaml#L26