FeatUp: A Model-Agnostic Framework for Features at Any Resolution

We thank the reviewer for their helpful comments.

Quantitative Ablations

We added ablations of every major architectural decision of FeatUp in the global response. Each ablation degrades performance across the metrics considered.

Justifying the focus of the paper

We present FeatUp as a framework that generalizes across backbones and upsamplers. The JBU and Implicit variants are our novel upsampler contributions that outperform existing upsamplers, but we do evaluate others. In particular:

• An important aspect of generality is the fact that FeatUp can work with any backbone network (see Fig. 6), making it applicable to many models in the community.
• Though we present our novel JBU upsampler that improves performance, Table 1 shows 4 other upsamplers can be trained with FeatUp including implicit networks, resize-convs, CARAFE [1], and SAPA [2]. These upsamplers are all trained with our proposed scheme (besides bilinear, strided, and large-image methods, which aren't trained). We note that our JBU variant, however, provides improvements over all other upsamplers.
• Though we provide 2 example downsamplers, we stress that this component is primarily dictated by “nature”. To draw a direct analogy with NeRF [3], the downsampler is analogous to ray-marching, which approximates the physics of light. In NeRF, this component can be modified slightly, (lens distortion, surface properties, light fields, etc) but needs to approximate physics. For FeatUp, Figure 6 shows that modern networks pool/blur information which leads us towards pooling/blurring operators for the downsampler. We also elaborate on this in the global response section “Clarifying the role of the saliency maps”. We have justified these downsamplers with extensive experiments, visualizations, and ablations.

Justifying the experimental settings

• We use existing experimental settings from the literature (FADE [4] + SAPA) for Table 2
• We compare against other per-image optimization methods including Deep Image Priors (DIP) [5] in Table 2 and Figure 8 and ZSSR [6] in Figure 8.
• Transfer learning is an established setting to measure feature quality. It is used throughout the unsupervised representation learning literature including in methods like STEGO [7], PiCIE [8], MOCO [9], SimCLR [10], DINO [11] and others. This analysis also shows we improve performance on downstream tasks without retraining downstream components, which is important for practitioners who want to upgrade or better understand existing methods.
• The parameters in the upsampler comparison of Table 1 are updated using the FeatUp training losses. We use these upsampling operators instead of JBU demonstrating that our framework works with a variety of upsamplers.

Role of JBU

Our paper focuses on making a system that can efficiently upsample features and retain their original semantics. We find that existing upsampling layers do not properly utilize guidance information as well as the JBU does. Our analysis shows our JBU operator beats all existing upsamplers for creating upsampled features, which is why we investigated it. We hope you can see this as an additional novelty of our work.

Explaining our relationship to “Fast End to End Trainable Guided Filter”

We thank the reviewer for bringing this work to our attention and have added it to our related work section. We note that “Fast End to End Trainable Guided Filter” [12] does not perform the full JBU computation but instead approximates it with learned pointwise convolutions and linear maps because the local query/model was computationally intractable. This is precisely the problem we solve exactly with our new efficient CUDA kernel.

Justification of FeatUp (Implicit) method

Please refer “Practicality of FeatUp (Implicit)” in the global response for advantages of any-resolution features. FeatUp (Implicit) is not the only per-image-optimized method as we compare to the widely-used approaches DIP and ZSSR that optimize results per image.

Performance discrepancy of SAPA

We appreciate you pointing this out. We have used SAPA’s official code, and use the encoder-decoder architecture and training objective detailed in SegFormer [13] and followed by FADE evaluations. All experiments are performed using the same controlled settings and environment to ensure a fair comparison between methods. We note this discrepancy the paper for full transparency.

Organization of Related Work

We have incorporated all suggested works into the related work. We evaluate some image super-resolution methods (DIP, ZSSR, and LIIF) in Section 6.3 which is why we included them in our related work. We shortened this section and directed readers to the supplement to provide more context. Regarding Amir et al. (2021) and Tumanyan et al. (2022), both works describe a modified patch embedding stride which is our “strided” baseline.

All details and results in this response have been added to our paper.

References

[1] Wang, Jiaqi et al. “CARAFE: Content-Aware ReAssembly of FEatures.” 2019 IEEE/CVF International Conference on Computer Vision (ICCV) (2019): 3007-3016.

[2] Lu, Hao et al. “SAPA: Similarity-Aware Point Affiliation for Feature Upsampling.” ArXiv abs/2209.12866 (2022): n. pag.

[3] Mildenhall, Ben et al. “NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis.” Commun. ACM 65 (2020): 99-106.

[4] Lu, Hao, et al. "FADE: Fusing the assets of decoder and encoder for task-agnostic upsampling." ECCV 2022.

[5] Ulyanov, Dmitry et al. “Deep Image Prior.” International Journal of Computer Vision 128 (2017): 1867 - 1888.

[6] Shocher, Assaf et al. “Zero-Shot Super-Resolution Using Deep Internal Learning.” 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (2017): 3118-3126.

[7] Hamilton, Mark et al. “Unsupervised Semantic Segmentation by Distilling Feature Correspondences.” ArXiv abs/2203.08414 (2022): n. pag.

[8] Cho, Jang Hyun et al. “PiCIE: Unsupervised Semantic Segmentation using Invariance and Equivariance in Clustering.” 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2021): 16789-16799.

[9] He, Kaiming et al. “Momentum Contrast for Unsupervised Visual Representation Learning.” 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019): 9726-9735.

[10] Chen, Ting et al. “A Simple Framework for Contrastive Learning of Visual Representations.” ArXiv abs/2002.05709 (2020): n. pag.

[11] Caron, Mathilde et al. “Emerging Properties in Self-Supervised Vision Transformers.” 2021 IEEE/CVF International Conference on Computer Vision (ICCV) (2021): 9630-9640.

[12] Wu, Huikai et al. “Fast End-to-End Trainable Guided Filter.” 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (2018): 1838-1847.

[13] Xie, Enze, et al. "SegFormer: Simple and efficient design for semantic segmentation with transformers." NeurIPS 2021

We thank the reviewer for their considerate response and for raising their rating to a 5. Could you please edit your original response to mark the improvement in score so that the system registers the change? We will upload another copy of the full text today that takes this additional round of feedback into account. Regarding your follow up points:

Especially when the framework is mainly used to justify the two upsampling operators.

The implicit version of the upsampler is justified moreso by analogy to NeRF and broad use in the literature. The JBU upsampler is also justified through an end to end semantic segmentation experiment, completely separate from the FeatUp losses and broader framework.

The authors also comment that "Table 1 shows 4 other upsamplers can be trained with FeatUp including implicit networks, resize-convs, CARAFE [1], and SAPA [2]". This confuses me. I have re-read the paper and the revision and find nowhere any text or training details stating how the comparing operators are trained with the proposed framework. .... As far as I know, resize-convs, CARAFE, and SAPA are just upsampling operators. Anyway, this is a place, which, I think, needs further clarification.

We adjusted the main text to make this training clearer. In the same way that multi-view consistency can be used to learn JBU parameters, we can use this same loss to learn parameters of SAPA, CARAFFE, or Resize Convs which all have trainable parameters that must be learned. Functionally, these upsamplers are identical to our JBU with just differences in their parameterizations and internal details.

In addition, I am also interested to know how a single upsampling operator is trained within the framework where a coupled downsampling-upsampling operator pairs are required.

Coupled upsamplers and downsamplers are not required. The downsampler acts as the ray-marcher does in NeRF, you can fit any kind of 3d representation (implicit, feedforward, etc) and just using ray-marching to get from 3d to 2d. In FeatUp any downsampler can be trained using either downsampler. Again, the downsampler is just an operation to learn the way deep networks pool semantics and is unrelated to the details of the upsampler. This is why we think of it as an ``independent'' component.

To justify the effectiveness and generality of the framework, in addition to the proposed two operators, a better choice may be to validate how existing coupled downsampling and upsampling operators work with the framework such as max pooling-max unpooling and space2depth-depth2space.

It is important to note that the upsampler and downsampler don’t need to be paired in FeatUp. Its more important to have a well parameterized downsampler so that it can capture the nonlinear and dynamic pooling behavior of a variety of modern deep networks. If after reading this, you still feel that the work needs these experiments we can start working on them.

If the proposed framework is training-free when used with downsampling tasks, this setting seems fine. But unfortunately, the framework still needs training with existing backbones. In this case, training the upsampling operators with a task-orientated loss and update the network parameters can mostly reveal the upper bound capability of the operators. I think this is the reason why existing work evaluates upsampling under an end-to-end setting as Table 2.

The important take away from table 1 is that you can directly improve the resolution of existing deep models without changing anything important about their deep features other than the resolution. This is critical if you want to use the high-res features of feat up without retraining other components of an architecture. It is also important if you want to study the behavior of existing models through CAM. The linear probe results also show that this resolution improvement fundamentally improves the quality of existing features as evaluated by the standard evaluation method used in hundreds of papers in the literature.

Also, I am curious to know how the framework works under the end-to-end setting in Table 2. Does it use the cylic loss and the task loss together? If this is true, the additional loss may be used as well on other comparing operators.

For the sake of making apples-to-apples comparisons we do not use our FeatUp losses to train the models in table 2, we only use the task loss.

The performance discrepancy of SAPA. If the authors find a diffculty in reproducing the results, a better way is to cite the results reported by the paper or to raise a issue at the online repository to ask why. Simply reporting an unmatched number seems inappropriate without any grounded explanation.

We will include both numbers and put an asterisk next to the literature reported one with an expanded description. We will also try to get in touch with the SAPA authors.

We thank the reviewer for their helpful comments.

Additional Experimental Details

We add all requested additional experimental details to our supplemental material. We provide hyperparameters for both FeatUp variants in Table 7 of the supplement, and provide backbone and training/validation data usage when a new experiment is introduced in the paper. We summarize important details below:

Experimental details - Table 2 (SegFormer)

When evaluating FeatUp as a drop-in replacement for upsampling operations in the SegFormer architecture, we use the encoder-decoder architecture and training objective detailed by Xie et al. [1] and followed by Lu et al. [2] for FADE evaluations. Specifically, we train SegFormer on ADE20k [3] (20,210 training images and 2,000 validation images) for 160k steps. To validate that our evaluation setup matches that of existing literature, we also compute FLOPs for SegFormer with various drop-in upsamplers and augment Table 2. These counts are comparable with those in [4], confirming that our architectural setup matches that of existing literature.

Experimental details - Table 1 (CAM, linear probes)

We provide details on the datasets, backbones, and feature resolutions for these evaluations in Section 6.12 of the supplement, but have revised the main section to also provide more details. Below, we provide these additional details:

• CAM scores: Average Drop and Average Increase are computed across 2,000 images from the ImageNet [5] validation set. These images are randomly selected (and kept constant across all evaluations) except for the criteria that the backbone’s classification of the image is equal to the ground truth label (otherwise the measures of Average Drop and Increase would be inaccurate). The FeatUp (JBU) upsamplers are trained on the ImageNet training set for 2,000 steps, and never see the validation images at training time. We use a frozen ViT-S/16 pre-trained in a supervised manner on ImageNet as the featurizer, and extract CAMs by applying a linear classifier after max-pooling ViT features. Upsampling is done (14x14 → 224x224) on the features themselves, and CAMs are obtained from these high-resolution maps.
• Linear probe (segmentation): We compute mIoU and accuracy over the COCO-Stuff validation set (27 classes) [6]. The FeatUp (JBU) upsamplers are trained on the COCO-Stuff training set for 2,000 steps, and never see the validation images at training time. We use a frozen ViT-S/16 pre-trained in a supervised manner on ImageNet as the featurizer, upsample the resulting features (14x14 → 224x224), and extract 27-channel segmentation maps by applying a linear layer on the features.
• Linear probe (depth estimation): We compute RMSE and $\delta$ > 1.25 scores over the COCO-Stuff validation set, with monocular depth outputs from the MiDaS v3 hybrid model [7] as ground truth. We use a frozen ViT-S/16 pre-trained in a supervised manner on ImageNet as the featurizer, upsample the resulting features (14x14 → 224x224), and extract 1-channel depth maps by applying a linear layer on the features.

Quantitative ablations

We thank the reviewer for bringing up the need for more rigorous evaluation of FeatUp. We have added quantitative ablations of both FeatUp variants on Table 1 (CAM, segmentation linear probe, and depth linear probe) evaluations, and ablations of FeatUp (JBU) on Table 2 (end-to-end SegFormer) evaluations. In all experimental settings, each ablation degrades segmentation performance, with the MLP exclusion generally being the most detrimental. See the global response for more details.

Code release

We agree with the reviewer that open-sourced code and pre-trained models are valuable assets to the community. Accordingly, we plan to release the code for both variants of FeatUp, its corresponding training code, and pretrained upsampling modules.

All details and new results in this response have been added to our paper revision (in yellow highlighted text).

References

[1] Xie, Enze, et al. "SegFormer: Simple and efficient design for semantic segmentation with transformers." NeurIPS 2021.

[2] Lu, Hao, et al. "FADE: Fusing the assets of decoder and encoder for task-agnostic upsampling." ECCV 2022.

[3] Zhou, B., et al. “Semantic understanding of scenes through the ade20k dataset”. IJCV 2019.

[4] Liu, Wenze, et al. "Learning to Upsample by Learning to Sample." ICCV 2023.

[5] Deng, Jia, et al. "Imagenet: A large-scale hierarchical image database." CVPR, 2009.

[6] Caesar, Holger et al.. "Coco-stuff: Thing and stuff classes in context." CVPR. 2018.

[7] Ranftl, René, et al. "Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer." IEEE PAMI.

We appreciate your thoughtful review of our work. If the rebuttal helped clear up things for you we kindly ask that you increase your score to give FeatUp a fighting chance. Thanks for your consideration!

We thank the reviewer for their helpful comments. We address questions and concerns below.

Segmentation Benchmarks

We evaluate commonly-used upsamplers on the SegFormer [1] architecture and the ADE20k [2] dataset, which is a standard semantic segmentation benchmark. As seen in Table 2, FeatUp outperforms the other upsamplers in mIoU, mAcc, and aAcc while remaining competitive in parameter count (it uses fewer parameters than every upsampler meant for features, with only A2U being lighter - however, A2U was designed for image matting and does not perform well on this task).

Comparison to additional references

We thank the reviewer for pointing out related prior work on feature upsampling, and have added them in our related work section.

We have also implemented “Learning Implicit Feature Alignment Function for Semantic Segmentation” [3], and added a visualization to Figure 8. While IFA performs well on specific semantic segmentation benchmarks, it does not take advantage of image guidance and fails to learn high quality representations outside of the encode-decoder framework.

Computation Cost Analysis

We analyze how GFLOPs (= FLOPs * 1e-9) scale for each upsampling operation in a new figure (see Figure 16 in the paper) and we add an analysis of FLOPS to Table 2. In general we find that FeatUp’s JBU operator is lightweight compared to other upsamplers and is small compared to the cost of evaluating the backbone network.

In particular, we study the original CARAFE [4], a “guided” CARAFE (the original operation with the guidance concatenated with the source to mirror the other guided operations in this study), SAPA [5], FADE [6], and FeatUp (JBU). Besides the attribute being varied (upsampling factor, feature dimension, or target spatial dimension), all other experimental settings are kept constant and exemplify a realistic feature-upsampling scenario (detailed below).

• First, we vary the upsampling factor. In each experiment, we begin with a $(1, 256, 8, 8)$ tensor, guide it with a $(1, 256, 8n, 8n)$ tensor when guidance is appropriate, and produce a $(1, 256, 8n, 8n)$ tensor with $n \in \{ 2, 4, 8, 16, 32 \}$.
• In the second plot, we vary the channel dimension. In each setup, we begin with a $(1, c, 8, 8)$ tensor, guide it with a $(1, c, 16, 16)$ tensor when guidance is appropriate, and produce a $(1, c, 16, 16)$ tensor with $c \in \{ 32, 64, 128, 256, 512 \}$.
• Finally, we vary the spatial dimension of the output. In each setup, we begin with a $(1, 256, s//2, s//2)$ tensor, guide it with a $(1, 256, s, s)$ tensor when guidance is appropriate, and produce a $(1, 256, s, s)$ tensor with $c \in \{ 16, 32, 64, 128, 256 \}$.

Note that the resize-conv operation’s GFLOPs can also be calculated in the same way, but when visualized on these plots, its computational load grows so quickly that the upsamplers depicted appear to have constant performance. Thus, we show the upsamplers that are most competitive in performance.

We find that FeatUp is competitive in GFLOP count to the current leading upsamplers and is lighter weight than FADE. It is comparable to CARAFE’s computational cost. These GFLOP analyses show that FeatUp (JBU) is not only architecturally and qualitatively a reasonable substitute for existing upsamplers, it is competitive in # FLOPs across a wide array of scenarios.

Importance of Any-Resolution Features

Please refer to the global response (“Practicality of FeatUp (Implicit)”) for descriptions of use cases for any-resolution features.

Importance of the Predicted Salience Map

Please refer to the global response (“Clarifying the role of the saliency map”) for an explanation of this design decision.

All details and new results in this response have been added to our paper revision (in yellow highlighted text).

References

[1] Xie, Enze, et al. "SegFormer: Simple and efficient design for semantic segmentation with transformers." Advances in Neural Information Processing Systems 34 (2021): 12077-12090.

[2] Zhou, B., Zhao, H., Puig, X., Xiao, T., Fidler, S., Barriuso, A., & Torralba, A. (2019). Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 127(3), 302-321.

[3] Hu, Hanzhe et al. “Learning Implicit Feature Alignment Function for Semantic Segmentation.” European Conference on Computer Vision (2022).

[4] Wang, Jiaqi et al. “CARAFE: Content-Aware ReAssembly of FEatures.” 2019 IEEE/CVF International Conference on Computer Vision (ICCV) (2019): 3007-3016.

[5] Lu, Hao et al. “SAPA: Similarity-Aware Point Affiliation for Feature Upsampling.” ArXiv abs/2209.12866 (2022): n. pag.

[6] Lu, Hao, et al. "FADE: Fusing the assets of decoder and encoder for task-agnostic upsampling." ECCV 2022.

References

[1] Ahn, Jiwoon et al. “Weakly Supervised Learning of Instance Segmentation With Inter-Pixel Relations.” 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019): 2204-2213.

[2] Song, Jiaming et al. “Denoising Diffusion Implicit Models.” ArXiv abs/2010.02502 (2020): n. pag.

[3] Mildenhall, Ben et al. “NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis.” Commun. ACM 65 (2020): 99-106.

[4] Ulyanov, Dmitry et al. “Deep Image Prior.” International Journal of Computer Vision 128 (2017): 1867 - 1888.

[5] Hamilton, Mark et al. “Unsupervised Semantic Segmentation by Distilling Feature Correspondences.” ArXiv abs/2203.08414 (2022): n. pag.

[6] Sitzmann, Vincent et al. “Light Field Networks: Neural Scene Representations with Single-Evaluation Rendering.” Neural Information Processing Systems (2021).

[7] Hu, Hanzhe et al. “Learning Implicit Feature Alignment Function for Semantic Segmentation.” European Conference on Computer Vision (2022).