MSPE: Multi-Scale Patch Embedding Prompts Vision Transformers to Any Resolution

Thank you for providing us with your valuable feedback and suggestions. We appreciate your input and have carefully considered your questions. Below, we provide detailed responses to each of them:

W1: Assuming $\mathcal{X} \sim \mathcal{N}(0, 1)$ in Image Processing

We completely agree with this point. FlexiViT aims to equalize patch embeddings, using the assumption $\mathcal{X} \sim \mathcal{N}(0, 1)$ for downscaling solutions. However, in real-world scenarios, image pixels do not necessarily meet this assumption, leading to poor FlexiViT performance at low resolutions.

In light of this issue, we introduce learnable patch embedding and propose a more reasonable optimization condition that minimizes loss across different resolutions, which could compensate the limitation of the above assumption. In other words, our method does not rely on $\mathcal{X} \sim \mathcal{N}(0, 1)$ assumption, and the experimental results in Sec. 4 demonstrate that our method significantly outperforms FlexiViT on low-resolution inputs. We would revise the motivation part (Sec. 2.4) to clarify it.

W2: Exploring Optimization of Encoder

We appreciate the reviewer's insightful question. In this study, we aim to apply minimal modifications to various pre-trained ViT models, finding that optimizing only the patch embedding layer achieves excellent performance. Indeed, optimizing the encoder can increase the model's performance but adds computational costs. Furthermore, optimizing the encoder must be handled carefully to avoid degrading existing representations and causing knowledge forgetting.

Optimizing the patch embedding layer does not impact the original model's performance, as MSPE only optimizes added convolutional kernel parameters. Below are the experimental results for optimizing the encoder, showing performance improvements at other resolutions but a decrease at the standard 224x224 resolution. We hope these results address your concerns about encoder optimization. We will add the above discussions in the revised version.

Q1: Testing on High-Resolution Images

Thank you for bringing up this important point. Below are our experimental results at higher resolutions, where we tested two types of ViT models, including Non-overlapping patch embedding (ViT) and overlap patch embedding (PVT). We will add the results of high-resolution in the revised version.

Thank you for your valuable feedback. We have carefully considered your questions and would like to address them as below:

W1: Experiments of Positional Encoding Strategies

Thanks for your comment. MSPE employs the vanilla position encoding approach, a learnable $(N, DIM)$ vector where $N = \left\lfloor \frac{h}{h_k} \right\rfloor \times \left\lfloor \frac{w}{w_k} \right\rfloor$ For non-standard input resolutions, we resize the $(N, DIM)$ vector to $(N_{new}, DIM)$ using linear interpolation, commonly used in ViT models as resample absolute position embedding.

Experiments in Sec. 4 demonstrate that, using the vanilla position encoding strategy, MSPE significantly outperforms other methods with complex position encoding strategies, such as ResFormer and NaviT. Actually, we intend not to use other complex positional encoding methods as they do not align with pre-trained ViT models. As shown in the results below, changing the position encoding significantly reduces the performance of pre-trained models (ViT/B-16), and retraining does not recover the expected performance.

W2: Analysis of Patch Embedding and Its Relative Importance

Thank you for this comment. We focus on improving patch embedding rather than the encoder or positional encoding for two reasons.

• First, as demonstrated by the analysis in W1, altering the positional encoding strategy necessitates retraining the trained ViT models, incurring significant costs.
• Second, modifying the encoder presents similar challenges. Our goal, as detailed in motivation (Sec. 2.4), is to adapt pre-trained ViT models to different resolutions with minimal changes.

Modifying patch embedding is easy to implement, compatible with pre-trained ViT models, and significantly improves performance at various resolutions without compromising the original performance.

• Does the analysis yield different conclusions for different tasks?

In Sec. 2.4, we examined the impact of patch embedding on classification tasks by analyzing the class token at various resolutions. In classification, the class token carries global semantic information and serves as the feature for classification. In segmentation and detection tasks, such as SETR and ViTDeT, the class token is similarly utilized as the feature map for segmentation and detection. Thus, the analysis of the class token is consistent in principle across these segmentation and detection models. The experimental results in Tables 5 and 6 support our findings, demonstrating performance improvement in segmentation and detection tasks.

W3: Analysis of Extended Training Epochs

Thanks for your suggestion. We have provided results for 1, 3, 5, 10, and 20 training epochs in Table 5 of Appendix B, which we hope will address your concerns. These results show that MSPE stabilizes after 5 epochs, and 20 epochs result in slight improvements without overfitting issues. One advantage of our method is that it requires only 5 training epochs, because MSPE only modifies the patch embedding layer of the ViT model and has few trainable parameters. Therefore, very few training epochs are needed.

W4: Minor Error

Thank you for highlighting this error. We agree with your suggestion about the kernel shape, and we corrected it in the revised version of the paper.

Q1, Q3: Analysis of $\lambda$ and $K$

• The loss function is defined as: $\mathcal{L}_{\theta}(x,y)= \sum _{i=1}^K \ell [(\text{Enc} _{\phi} (g _{\widehat{\theta} _i} (B _{r}^{r _i}x)), y ] + \lambda \cdot \ell [(\text{Enc} _{\phi}(g _{\theta}(x)), y ]$ $\lambda$ is a hyperparameter to prevent performance degradation. The ablation study in Figure 7(a) presents results for $\lambda$ values of 0, 1, and 2, indicating: (1) The parameter is essential for maintaining performance, with $\lambda=1$ enhancing accuracy by 4.2% over $\lambda=0$. (2) The hyperparameter shows limited sensitivity, as increasing from $\lambda=1$ to $\lambda=2$ makes little difference. The revised version will include a wider range of $\lambda$ results.

• $K$ represents the number of convolutional kernels used for patch embedding. It is impractical to use a unique kernel size for each resolution. In MSPE, both the size and ratio of kernels $(w_{\theta}^i, b_{\theta}^i)$ are adjustable. The ablation study in Figure 7(b) demonstrates that $K=4$ is adequate, employing kernel sizes of 16x16, 12x12, 8x8, and 4x4.

Q2: Differences between MSPE and FlexiViT

FlexiVit optimizes patch embeddings consistently across resolutions through explicit analytical transformations. However, it has two disadvantages:

• First, it imposes a strict sufficient condition for handling varying resolutions, but the similarity in patch embeddings does not ensure optimal performance.
• Second, in the downscaling scenario, there is no analytical solution for this goal.

Differently, MSPE optimizes patch embeddings by minimizing the objective function across different resolutions, employing multiple dynamically adjustable patch embedding kernels to handle different resolutions effectively. Experimental results demonstrate that MSPE significantly outperforms FlexiVit.

Q4: Test at Non-Standard Resolutions

In Figures 1, 4, and 9 of our paper, we tested many uncommon resolutions, such as 192x144, 80x192, 896x128, etc. Our method can also be applied to resolutions like 200x150.

The response has resolved my concerns, and I have raised the score.

Thank you for providing valuable feedback and suggestions. We greatly appreciate your input and completing the necessary experiments. Here is our detailed response to your questions:

Q: Experimental Results on More Than 10x Resolution

Our method can be directly applied to high-resolution images because the shape and size of the patch embedding kernels dynamically adjust according to the input resolution and have multiple convolution kernels corresponding to different resolution ranges. Below are our experimental results for high-resolution images. We tested models with non-overlapping patch embedding (ViT) and overlapping patch embedding (PVT). We will include those results in the revised version.

We greatly appreciate your insightful comments and suggestions, as they have been helpful in refining and enhancing our work. We have thoroughly reviewed all of your points and have addressed your concerns as outlined below:

W1: Technical Contribution

Thank you for your kind feedback. In fact, images of different sizes and aspect ratios represent different data distributions, and effectively handling these varying resolutions is critical to addressing data distribution challenges in open-world settings. This is particularly crucial for real-world applications, as image resolutions are diverse, not fixed. However, ViTs lack robustness to different resolutions. As demonstrated in our experiments (Sec. 4.1), a model used for a 224x224 resolution significantly underperforms when processing inputs at 448x448 or 128x128 resolutions.

Previous works faced two major issues: (1) introduced complex modules and strategies that are incompatible with existing ViTs architectures, requiring the entire model to be retrained; (2) insufficient adaptability across various sizes and aspect ratios, limiting their performance to multiple resolutions. Our method does not require costly training or complex modifications, making it easily applicable to most ViT models. Moreover, it significantly outperforms previous methods. The above discussion will be added in the revised version.

W2: Scalability Concerns

We agree that scalability is crucial for method applicability. In Sec. 4 of our paper, we conducted extensive experiments across multiple scenarios and datasets, including ImageNet-1K, COCO2017, ADE20K, and Cityscapes, as well as on various ViT models like ViT, DeiT III, and PVT. To further address your concerns about scalability, we also performed high-resolution tests on ImageNet-1K. The following experimental results demonstrate that our method performs excellently across large datasets and high-resolution settings. We will add those results to the revised version.

W3: Ablation Studies

Thank you for your suggestions. We conducted extensive ablation experiments in Sec. 4.4 and Appendix D, including training epochs, model size, hyperparameters, kernel count (K), and resizing methods. The results are displayed in Figures 6, 7, 8, 10, and Table 6. The experiments show that our method is robust to hyperparameters and that increasing training epochs and kernel counts boosts performance. We achieve good performance and basic convergence with 5 training epochs and 4 kernels. We also evaluated different resizing methods, including nearest, bilinear, and bicubic, with PI-resize proving the most effective, as shown in Figures 6 and 8. We hope these detailed results address your concerns about ablation studies. We would revise the ablation study part (Sec. 4.4) to make it clearer.

W4: Real-World Applications

We completely agree with this. Adapting to different resolutions is a key challenge for ViT models in real-world applications. Our method was evaluated on multiple real-world datasets such as Cityscapes, ADE20K, ImageNet-1K, and COCO2017, including tasks like classification, detection, and segmentation. The City dataset collects street scenes from various cities; ADE20K includes scenes from more than 150 settings; ImageNet-1K and COCO2017 are extensive, covering numerous real-world environments.

Tables 5 and 6 in our paper show that MSPE significantly improves across various resolutions and task settings. We hope these experiment results confirm the substantial potential for real-world application.