LR0.FM: Low-Res Benchmark and Improving robustness for Zero-Shot Classification in Foundation Models

W1: Although I agree with.... LR0.FM benchmark.
&
Q1: Is the low-resolution performance .... practical use case on it?

One of the reasons zero-shot / open-vocabulary models are gaining popularity is because of their real-world application (e.g. YOLO-WORLD [1]). But the real world is noisy and has low-resolution images, which is where the need for robustness arises. 16x16 and 32x32 do not only represent the image sizes (small face, far away galaxies) [Line 52 & 69] but also represent pixelation, i.e. enlarging 16 x 16 to 224 x 224 (or higher resolution) mimics far away objects like HQ surveillance camera footage for far object [Line 192-194], or privacy protected data (Line 52). Figure 29, contains some real-world examples where the model makes simple mistakes in recognizing images if the images contain low-resolution artifacts like pixelation.

[1]: Cheng, Tianheng, et al. "Yolo-world: Real-time open-vocabulary object detection." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

---

---

W2: There is no comparison.... or has a unique pattern.

To the best of our knowledge, there isn't any related work for low resolution in zero-shot conditions for foundation models (FMs/VLMs) [Line 150-151].
The closest work to ours is Robust SAM [1], which tries to improve the robustness of the Segment Anything Model (foundation model) against noisy images. We have tried modifying the segmentation model to our classification task (denoising modules, and trainable prompts) in Table 4. Real-world benchmarking results that overlaps with some of our findings include:

1. Noisy Image Segmentation masks [1]: Preserving original weights helps develop robustness (LR-TK0), and Fine-tuning drops the zero-shot capabilities (Figure 7 left).
2. ImageNet-1K pretraining may be more robust than LAION-2B pre-training (in case of fine-tuning). [2] (Figure 6 left).

[1] Wei-Ting Chen, Yu-Jiet Vong, Sy-Yen Kuo, Sizhou Ma, and Jian Wang. Robustsam: Segment anything robustly on degraded images. 2024.
[2] Hwang, Jaedong, et al. "ImageNet-RIB Benchmark: Large Pre-Training Datasets Don't Guarantee Robustness after Fine-Tuning." arXiv preprint arXiv:2410.21582 (2024).

---

---

W3: LR-TK0 is a valid way .... enhancing the contribution of LR0.FM.

Our proposed LR-TK0 is dependent on LR0.FM in some fundamental ways:

1. Robustness (evaluation metric) improvement is derived using the benchmarking of 66 models (WAR metrics)
2. Knowledge of (Figure 7 (right) indicates that the lower / shallow layer deviates from HR model layers more than the deeper layer. This contradicts traditional LR-based techniques that deploy trainable modules on top of the final features, focusing more on the deeper layers than the initial ones. [Line 373-374]
3. Knowledge of Figure 2, which indicates models make semantically correct predictions on low resolution, is the motivation for preserving pre-trained weights.

Additionally, compared to the research in low resolution (far more comprehensive), our solution is very basic and intended to motivate future research in this direction. Existing techniques like denoising module [1], inverse LR training module [2], and specialized resolution modules [3], if deployed on LR Tokens will likely see a far superior performance gain. Our solution is a basic setup for their solutions.

[1] Wei-Ting Chen, Yu-Jiet Vong, Sy-Yen Kuo, Sizhou Ma, and Jian Wang. Robustsam: Segment anything robustly on degraded images. 2024.
[2]: Lu, Yuhang, and Touradj Ebrahimi. "Cross-resolution Face Recognition via Identity-Preserving Network and Knowledge Distillation." 2023 IEEE International Conference on Visual Communications and Image Processing (VCIP). IEEE, 2023.
[3] Chai, Jacky Chen Long, et al. "Recognizability embedding enhancement for very low-resolution face recognition and quality estimation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

---

---

Thank you for your response.

It addressed my questions.

I believe weaknesses 1 and 3 cannot be fully resolved through rebuttal comments. However, despite these weaknesses, the paper offers valuable contributions.

I will maintain my initial rating.

For W2 and Q2, I hoped to see benchmark comparisons similar to [1]. Including such comparisons across various benchmarks, as in [1], could enhance the paper's contribution.

[1] Do Better ImageNet Models Transfer Better?, CVPR 2019

W5: In section 3, in ....... M is the size of dataset.

Thank you for bringing these to our attention. The revised version now reads; [Line 175-177]
Backbones are referred to using their publicly available pre-trained weights, e.g. CLIP-ViT L (400M), which means: CLIP model ViT-L architecture, pre-trained on 400 million datasets.

---

---

W6: In evaluation metrics, in $\gamma_n^D$.... m=224,256,378… is better.

Thank you for the constructive suggestion, it's now included in the revised version at [Line 195-197].

---

---

W7: There are some.... no (a) and (b) in the figure

Thank you for bringing these to our attention. All corrections are added to the revised copy. The final copy has replaced (a) with “left” and (b) with “right” in Fgure 10 caption.

---

---

W8: In table 5, what is .... next below row?

Apologies for the confusion. The baseline row now reads as ‘Baseline (frozen)’ which is pretrained EVA-B/16 zero-shot performance. The next row is “not” frozen, (end-to-end finetuning), “doesn't” have LR-tokens (our contribution), and “doesn't” have a task-specific classifier (similar to our technique). [Line 481-483].

---

---

W9: In Grad-CAM results, it's not clear what the vanilla model is.

Thank you for bringing these to our attention. Figure 16 (& caption) now reads baseline instead of vanilla, where EVA-B/16 is the baseline model [Line 431].

---

---

W10: The figure caption should....as Figure 13 and Figure 14.

Thank you for bringing these to our attention. All captions are updated to include more descriptions.

---

---

W11: In some figures, it's not needed to use arrow, like in figure 4 and figure 5.

Thank you for bringing these to our attention. The arrows have been removed in the final copy.

---

---

We hope that we have resolved all the issues raised by the reviewer. We are happy to answer any more doubts and would thank you once again for the feedback.

W1: In Problem B) SAR ... understand why WAR is proposed.

Apologies for the confusion regarding the proposed metrics. We will try to simplify the “Problem B of SAR overlooks dataset” and have updated the simplification to the revised copy as well [Line 248-269]. We will try to answer the concerns in parts

Once robustness is calculated (using traditional $\gamma_{n}^{D}$ or our improved one $\Gamma_{n}^{D}$), a common solution is to average scores across the datasets (giving each dataset a score of 1) this is called SAR, Simple Aggregated Robustness [Line 198-200]. SAR is a simple averaging of robustness scores across all the datasets.

1) Ordering of the model
Once the robustness scores are aggregated across the dataset, the standalone aggregated scores don't mean anything unless we are comparing models. The ranking of the models after aggregation across the datasets compares the models. [Line 247-249]. Ideally, model ranking, after averaging robustness across datasets, should stay consistent with their rankings on individual datasets. This consistency of ranking is measured by Spearman Rank correlation (measures relative ordering of two distributions).

Middle subfigure of Fig 4
Whether we use traditional relative robustness $\gamma_{n}^{D}$ (abnormally high value) or our improved robustness $\Gamma_{n}^{D}$ (Problem A solution), the SAR ranking of models is uncorrelated / weakly correlated with EuroSAT ranking of models (0.26) and moderately correlated with ImageNet-A (0.56) [Line 249-251]. WAR adjusts the dataset weights so that the model rankings after aggregation reflect each dataset fairly (fig. 4 (right)).

2) Objective function of AX optimization
AX optimization is a hyperparameter optimization tool that maximizes a certain objective. In our case, adjust the weights of the datasets (hyperparameters) with the overall goal of maximizing the Spearman Rank correlation between the final model ranking obtained after the weighted averaging of datasets and individual dataset ranking on empirically found objective function (equation 2) $0.95 \times SC(Imagenet) + 0.95 \times SC(ImageNet-V2) + 0.95 \times SC(DTD) + SC(ImageNet-A) + SC(EuroSAT)$

Figure 4 indicates this via a spider curve.

1. WAR increases the correlation score of EuroSAT (0.26 → 0.49) from weakly / no correlation to moderate correlation, and ImageNet-A (0.56 → 0.68, strong correlation is 0.70).
2. For highly correlated datasets small difference between SAR and WAR is ideal i.e. WAR preserves the high correlation scores of these datasets.

---

---

W2: In Table 2, Ours .... why ours is more effective.

We apologize for the confusion and Table 2 caption is updated to reflect LR-TK0 improvement on various Foundation models. Table 2 is not a comparison with the baseline, hence “fairness” is not applicable. We have tried to show the gain in robustness by applying our LR-TK0 technique to the baseline foundation models.

There are two novelties here [Line 360-364]:

1. We are trying to increase the robustness of the models against LR images, without discarding the pre-trained weights with minimal parameter gain. Table 2: 224 x 224 accuracy indicates we are pretty much retaining the initial zero shot accuracy (1-2% accuracy drop). For LR 16 x 16, the accuracy jump is increasing from (4-9%) across different models.
2. We are not training/fine-tuning on any of our target datasets, making it data-free (synthetic data) (Unlike previous methods, RobustSAM [1]). Our goal is to propose a “zero-shot” technique for increasing robustness against LR.

Table 3 SR methods & Figure 9: Both intend to show SR (super-resolution) methods don't work well in zero-shot conditions, i.e. they need target datasets to improve performance, which defeats the zero-shot aspect of LR robustness. Our method trained on synthetic datasets doesn't require finetuning on any of the target datasets.

[1] Wei-Ting Chen, Yu-Jiet Vong, Sy-Yen Kuo, Sizhou Ma, and Jian Wang. Robustsam: Segment anything robustly on degraded images. 2024.

---

---

W3: In the introduction, ... the existing metrics are*.

Thank you for the suggestion. The revised version now reads; “Metrics for measuring robustness ($\gamma$, Schiappa et al. (2024)) and its averaging across datasets (SAR) have some limitations;“ [Line 78-80]

---

---

W4: In Figure 1, it's not clear what the y-axis is.
Apologies for the confusion. The y-axis is zero shot classification accuracy and the x-axis is the resolution. Figure 1 caption now reflects this.

Dear reviewer,

Thank you once again for your time in reviewing our paper and providing valuable feedback. As the discussion period extended till 2nd December, we are reaching out to see if you have any further questions or pending issues.

We have hoped to resolve your concerns with WAR-SAR evaluation metrics, and clarity in writing.

Please let us know if you have any follow-up comments or require additional clarifications.

Thanks for the detailed response.

Please update the final version accordingly.

The rating has been updated from 3 to 6.

W1: The "misleading high robustness" .... 5) How to choose the hyperparameter alpha in Eq. 1?

Apologies for the confusion regarding the proposed metrics. We will try to simplify the “Problem B of SAR overlooks dataset” and have updated the simplification to the revised copy as well [Line 248-269].

4) Confusion regarding SAR
We have 15 datasets, one model may be robust on 1 dataset and worse on the other. This robustness is measured by traditional relative robustness $\gamma_n^D$ and can give abnormally high values for certain models on certain datasets. (Problem A, [Line 202-206]). Problem A solution is improved relative robustness represented by $\Gamma^{D}_{n}$ that won't give an abnormally high value.

Once robustness is calculated (traditional $\gamma_n^D$ or our improved one $\Gamma^{D}_{n}$), how do we compare the robustness of the two models? Dataset by dataset basis is not feasible. A common solution is to average scores across the datasets (giving each dataset a score of 1) this is called SAR, Simple Aggregated Robustness [Line 198-200]. SAR is not dataset-specific, but a simple averaging of robustness scores across all the datasets.

1) What does the ordering of the model mean
Once the robustness scores are aggregated across the dataset, the standalone aggregated scores don't mean anything unless we are comparing models. The ranking of the models after aggregation across the datasets compares the models. [Line 247-249]. Ideally, model ranking, after averaging robustness across datasets, should stay consistent with their rankings on individual datasets. This consistency of ranking is measured by Spearman Rank correlation (measures relative ordering of two distributions)

3) middle subfigure of Fig 4
Whether we use traditional relative robustness $\gamma_n^D$ (abnormally high value) or our improved robustness $\Gamma_n^{D}$ (Problem A solution), the SAR ranking of models is uncorrelated/ weakly correlated with EuroSAT ranking of models (0.26) and moderately correlated with ImageNet-A (0.56) [Line 249-251]. Figure 4 middle doesn't show WAR metrics. $\Gamma^{D}_{16}$ is the “improved robustness metrics”, (solution to Problem A). WAR adjusts the dataset weights so that the model rankings after aggregation reflect each dataset fairly (fig. 4 (right)).

2) Objective function of AX optimization
AX optimization is a hyperparameter optimization tool that maximizes a certain objective. In our case, adjust the weights of the datasets (hyperparameters) with the overall goal of maximizing the Spearman Rank correlation between the final model ranking obtained after the weighted averaging and individual dataset ranking on empirically found objective function (equation 2). $0.95 \times SC(Imagenet) + 0.95 \times SC(ImageNet-V2) + 0.95 \times SC(DTD) + SC(ImageNet-A) + SC(EuroSAT)$

5) How to choose alpha
Alpha is the rate at which robustness declines as accuracy approaches random prediction. Any alpha >> 1 is a good choice, (Figure 21). We chose 200 as a middle between 100 (the drop starts at 0.2) and 500 (the drop starts very close to 0). [Line 246-247]

---

---

W2: The figures are hard to.....such as Fig. 3 to Fig. 5.*

We apologize for the lack of clarity in the figure. As a revision, we have increased the spacing between figures for easy readability.

1. Figure 5 (right) i) size shows GFLOPs non-correlation with robustness, but model size correlates with robustness. Figure 5 (right) ii) size indicates model size (positively impact robustness).
2. Figure 7 (mid) The length of the bar is indicative of the average robustness of (66 models) for the dataset.
3. Figure 3 word cloud is generally used in benchmarking papers for summarizing datasets, (also used in Elevater [1])
4. Subfigures (different goals) save us white space that would be created with individual figures.

[1] : Elevater: A benchmark and toolkit for evaluating language-augmented visual model (Nuerips’22)

---

---

W3: The paper claims that 66 ... What are they?

The revised version contains the explicit mention of the 66 backbones in Table 1 caption. Table 1 provides a comprehensive overview of all the foundation models considered for this study and their corresponding backbones (66 in total). For example (in row-1 of Table-1), the CLIP model has four Vision Transformers (ViTs) and five ResNet backbones.

W4: The LR-TK0 technique ... three numbers in Fig. 10?

We apologize for the lack of clarity on certain terminology.

1. LR tokens are just ordinary trainable transformer tokens, indifferentiable from any other transformer tokens added on top of spatial tokens [Line 367-370]. Papers often name these “newly added tokens”, for example, “Class token in ViT” [1], “Distillation token” in Deit [2], “Prompts” in VPT [3].
2. There is no “Path generation", but “Patch Generation” in Figure 10. Patch Generation is the stage before transformer blocks where RGB image patches are converted into patch tokens.
3. Apologies for the unclarity in the original figure. (Figure 10 right) now has been fixed to show two contrastive losses which takes two embeddings to align them contrastively.

[1]: Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. International Conference on Learning Representations, 2021.
[2]: Touvron, Hugo, et al. "Training data-efficient image transformers & distillation through attention." International conference on machine learning. PMLR, 2021.
[3]: Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference on Computer Vision, pp. 709–727. Springer, 2022.

---

---

W5: The experiment analyses of .... lacks a summary of Table 3.

We apologize for the non-clear description.

Table 2
Table 2 caption explicitly mentions LR-TK0 for different foundation models. It doesn't have SR (super-resolution) methods, nor our method LR-TK0 (as shown in Figure 10) uses any super-resolution. Table 2 has baseline foundation models and improvement via our LR-TK0 technique. [Line 467-471] describes the analysis of Table 2, indicating consistent enhancement of robustness at low resolutions (16×16 & 32×32), particularly for MetaCLIP. While the low resolution is often seen as a domain shift problem [1], leading to potential declines in HR performance, our multi-scale training and HR teacher distillation minimize accuracy drops at higher resolutions.

Table 3
[Lines: 472-476] describes “Table 3 compares EVA-B/16 with super-resolution (SR) methods, with SR methods performing poorly in zero-shot settings for very low resolutions (fig. 9). In contrast, our approach is better suited for zero-shot scenarios. Diffusion-based SR method like IDM is too computationally expensive to evaluate on large datasets like ImageNet (results in Supplementary (Section F2 and F3))”. Regarding the experimental settings of the SR method in Table 3, we provide the details along with a code snapshot in the supplementary Section-E [lines: 1137-1174].

[1] Efficient Low-Resolution Face Recognition via Bridge Distillation. IEEE Transactions on Image Processing, 2020.

---

---

W6: Some minor typos: ... SAR overlooks datasets".

Thank you for bringing these to our attention. All corrections are made to the revised version.
We have followed the convection: If a sentence starts with word "Figure" (capital F) and if it mentions it in the middle of the sentence "fig." (small f).
We have reworded the naming convention for ease of understanding [Line 175-177]:
Backbones are referred to using their publicly available pre-trained weights, e.g. CLIP-ViT L (400M), which means: CLIP model ViT-L architecture, pre-trained on 400 million datasets.

---

---

We will be happy to answer any more doubts and would thank you once again for the feedback.

Dear reviewer,

Thank you once again for your time in reviewing our paper and providing valuable feedback. As the discussion period extended till 2nd December, we are reaching out to see if you have any further questions or pending issues.

We have hoped to resolve your concerns with WAR-SAR evaluation metrics, and clarity in writing.

Please let us know if you have any follow-up comments or require additional clarifications.

Q1: I am not sure to have.... choice of LR tokens position.

Apologies for the confusion, caption (figure 7) and [Line 347-352] are updated to reflect this: Figure 7 right Layers-wise similarity (e.g. 16 x 16 model layers with 224 x 224 ones). The lower right half indicates the similarity of deeper layers (brighter means more similar), while the upper left represents shallow layers (dull means less similar).
This means the proposed solution should focus on the initial layers as much as the later final layers. Since HR features are matched with LR features (trainable modules are traditionally introduced after the final features), this inherently gives more weightage to the final layers than the initial layers. Introducing LR tokens at every block treats each layer the same (especially shallow layers). Figure 15 performance validates this.

---

---

Q2: Additionally, in Figure 15, ....starting from the 5th block?*

Apologies for the confusion, the caption is updated to reflect this: “[i] LR tokens introduced starting from i-th block (& none after patchification)”, which would mean the LR tokens were introduced starting from the i-th block onwards (5th block in your example). [Line 521-524] describes this as well.

---

---

Q3: Potential typos: Thank you for bringing these to our attention. All corrections are made to the revised copy (uploaded)

---

---

We will be happy to answer any more doubts and would thank you once again for all the feedback and your time.

W1: The authors mention that.... token learning process.

Thank you for the insightful remark. To better resolve this issue, we would answer this, in parts:
Observation on “real” low-resolution images and degradation at 64 x 64.
The reviewer is correct in pointing out that real-world low-resolution images may behave differently from the ones generated synthetically via bicubic interpolation.

1. To conduct detailed analysis/benchmarking, we need HR - LR image pairs to measure robustness (drop in accuracy), measure gain in robustness (performance of LR-TK0), and the exact resolution at which accuracy starts dropping (64x64 in this study). To the best of our knowledge, there are no such real-world image classification datasets with cross resolutions or HR-LR pairs. It will be very useful to explore further if such a dataset is available in the future.
2. We did a small-scale analysis of real-world low-resolution images, to validate that the models make “semantically reasonable predictions” and “simple mispredictions” on such images (Figure 29) and found that the models behave like the synthetic ones (Figure 2 and Figure 28).
3. There are works that have seen a correlation between natural distortions and synthetic corruption [1]. Additionally, existing works [2] on low-resolution have traditionally used bicubic interpolation to simulate low-resolution and use it to compare various techniques for improving robustness.

Analysis and Proposed Technique for Real-world Images
Existing real-world benchmarking that overlaps with some of our findings includes:

1. Noisy Image Segmentation masks [3]: Preserving original weights helps develop robustness (LR-TK0), and Fine-tuning drops the zero-shot capabilities (Figure 7 left).
2. ImageNet-1K pretraining may be more robust than LAION-2B pre-training (in case of fine-tuning). [4] (Figure 6 left).

[1] Michaelis, Claudio, et al. "Benchmarking robustness in object detection: Autonomous driving when winter is coming." arXiv preprint arXiv:1907.07484 (2019).
[2] P. Li, L. Prieto, D. Mery and P. J. Flynn, "On Low-Resolution Face Recognition in the Wild: Comparisons and New Techniques," in IEEE Transactions on Information Forensics and Security, vol. 14, no. 8, pp. 2000-2012, Aug. 2019, doi: 10.1109/TIFS.2018.2890812.
[3] Wei-Ting Chen, Yu-Jiet Vong, Sy-Yen Kuo, Sizhou Ma, and Jian Wang. Robustsam: Segment anything robustly on degraded images. 2024.
[4] Hwang, Jaedong, et al. "ImageNet-RIB Benchmark: Large Pre-Training Datasets Don't Guarantee Robustness after Fine-Tuning." arXiv preprint arXiv:2410.21582 (2024).

---

---

W2: The phenomenon shown in Figure 2 .... low-resolution (relating to first point)?

We agree that the reasonable semantic predictions are very interesting and confirm that it occurs very frequently at low resolution. More examples are added in Supplementary Figure 28. Real-world low-resolution images are shown in Figure 29. which indicates 1) simple mispredictions, 2) semantically correct predictions, and 3) correct predictions.

---

---

W3: The LR-TK0 framework is .... model for ''intermediate'' resolutions?

In order to better resolve this issue, we will answer it in parts
Par or lower than the original model
The reviewer is correct in pointing out that increasing robustness on low resolution comes with a tradeoff drop in performance on higher resolution. In the main paper [Lines 468–469], we discussed that this saying decline in higher-resolution performance is often attributable to a domain shift problem, as highlighted in prior work [1]. For our benchmarking, performance drop starts below 64 x 64, thus have mainly focused on resolutions 32 x 32 and 16 x 16. [Line 191-193].

Compromising initial performance

However, 224 x 224 accuracy indicates we are pretty much retaining the initial zero shot accuracy (1-2% accuracy drop). For LR 16 x 16, the accuracy jump is increasing from (4-9%) across different models. It is also noteworthy that other adopted methods in our settings such as Visual Prompt Tuning (VPT) [2] and RobustSAM [3] face similar challenges. Despite this, our framework demonstrates performance closest to the baseline (Table 4),

[1] Efficient Low-Resolution Face Recognition via Bridge Distillation. IEEE Transactions on Image Processing, 2020.
[2] Visual Prompt Tuning. ECCV 2022.
[3] RobustSAM: Segment Anything Robustly on Degraded Images. CVPR 2024.

I would like to thank the authors for their detailed reply.

I see the limitation of not having datasets of real LR-HR images pairs. I still believe that the experiments with downsampled images are extensive and that it's reasonable to suppose the conclusions would correlate with natural LR images.

Thanks you for adding more examples in Figure 28. For Figure 29, I'm a bit puzzled about the significance of the observation; I agree the predictions are semantically reasonable, however 4 out of 6 images are well-classified, which means LR isn't a problem for these examples. I believe more interesting examples would be those where the top-1 prediction is incorrect though semantically reasonable. It would be also interesting to report the resolutions of images in Figure 29.

My other concerns are addressed.

I've read other reviewers' comments and i would like to thank them for bringing up details which were clarified by the authors in the revised paper. The paper is heavy in terms of experiments, insights and figures so it's crucial that everything is clearly detailed.

I'm still a bit puzzled about this sentence: "Ideally, the model rankings, after averaging, should stay consistent with individual dataset rankings". What does "individual dataset rankings" means? Do the authors mean: the model rankings, after averaging their accuracy across the datasets, should stay consistent with their rankings for individual datasets?

Thanks to the authors for their reply.

I won't linger more on the point about Figure 29. I understand the challenge of real low-resolution and i see how resizing can be misleading.

Remark: I wrote erroneously accuracy instead of robustness in my suggested reformulation (typo). Sorry for that. It's clear in the paper that it's robustness score, sure!

Overall, I believe there is no ''seriously critical'' point in the paper. Given the detailed rebuttal by the authors and the valuable contribution of the paper, I am happy to increase my score from 6 to 8 :)