Scaling Laws in Patchification: An Image Is Worth 50,176 Tokens And More

We sincerely thank the reviewer for their careful evaluation and thoughtful comments. Detailed responses can be found below.

Q1: Experiments with other architectures (e.g., Swin Transformer, LV-ViT)

Thank you for your insightful feedback. We would like to highlight that our conclusion also holds for pyramid networks such as Swin Transformer (see results in the response of Q1 for Reviewer DwN7) and stronger ViT baselines such as LV-ViT (see table below). To complete this experiment within a limited time frame, here we use 112×112 input images and employ LV-ViT-S as the backbone. Under this stronger baseline, we observe a performance trend similar to that in the standard ViTs, which demonstrates the architecture-wise robustness of patch size scaling. We will include the corresponding discussion in the revised version.

Q2: Comparison with early convolutions to the plain ViT model.

Thank you for your suggestion of comparing with early convolutions. We summarize the results of patchification scaling with early convolution in the table below. As shown, when the patch size is at the standard 16×16 level, applying early convolution brings a noticeable performance gain. However, this benefit diminishes rapidly as the patch size decreases, and becomes negligible at 8×8. In simple terms, the basic idea behind early convolution is to decompose a single 16×16 kernel size and 16×16 stride convolution layer into a stack of smaller kernels (e.g., 3×3) with smaller strides (e.g., 1×1 or 2×2). This approach effectively addresses the training instability caused by the abrupt spatial resolution drop at the patchification layer in standard ViTs. However, as the patch size decreases, this issue becomes much less pronounced. We are encouraged to see that our paper shares some core insights with this line of research—namely, that large-kernel patchification in standard ViTs can limit model expressivity. The difference is that we take a more direct approach to reduce the spatial downsampling effect in patchification. When extended to the extreme case of pixel tokenization, this issue is already fundamentally resolved. We will include these results and discussions in the revised version.

Q3: Could these two types of scaling methods (patch size and parameter size) benefit from each other?

Thank you for your insightful question. Yes, they do benefit each other and we have related results in Table 4 and discussions in line 364-376r. Table 4 shows a consistent upward trend from the top-left (small model and large patch size) to the bottom-right (large model and small patch size) corner. Additionally, as observed from both Table 1 and Table 4, reducing the patch size and increasing the input resolution can both have a positive impact, demonstrating the good potential of jointly employing these scaling dimensions.

Q4: Application to Mamba-like models.

Thank you for your feedback. We evaluate the patch size scaling performance with ViT and Adventurer models, where for Adventurer, we are actually using its Mamba-based setup so the application to Mamba models is already included. We will clarify this point in the revision.

Q5: I was supposed that a novel method is presented to solve the computation overhead when using smaller patches. If such a method is proposed, I think the overall quality of this paper could be further improved.

We appreciate your constructive comment on this point! In fact, we first identified a solution to address the computational challenges brought by small patch sizes before conducting the patchification scaling study: we chose to carry out the main experiments using the Adventurer model, whose computational cost scales linearly—as opposed to ViT’s quadratic scaling—with respect to sequence length. This linear complexity fundamentally resolves the computation bottleneck, which enables us to perform pixel tokenization experiments using modest computational resources (256 A100 GPUs). We also discuss the advantages of this linear architecture in Table 6: in the most demanding training setup with the longest sequences, Adventurer achieves an 8.4× speedup compared to ViT with FlashAttention. This substantial efficiency gain is what made our large-scale pixel tokenization experiments practically feasible.

We sincerely appreciate your constructive review, insightful feedback, and recognition of the empirical contribution of this work. The detailed response is summarized below.

Q1: No technical or theoretical contribution.

Thanks for the comment. As you noted, this is indeed an experiment-driven study, and our focus is on delivering empirical results to demonstrate a new scaling law. We respectfully ask the reviewers to take into consideration that all studies on scaling laws are inherently grounded in empirical observations. For large vision or language models, it is extremely difficult—if not impossible—to theoretically derive precise upper or lower bounds on their fitting capacity. Therefore, empirical evidence remains the most practical and informative approach for uncovering such trends.

If we consider the initial scaling law proposed by Kaplan et al. (2020) as a theoretical foundation, this work will be a meaningful extension of their theories in vision. Specifically, in the initial study, they show that in language modeling tasks, when the model size (parameter count $M$) and the amount of training data (token count $N$) increase, the perplexity (or loss) on the validation set tends to follow a near power-law relationship:

$L(N,M)\approx\alpha N^{-\beta}+\gamma M^{-\delta}+\text{(smaller interaction terms)}$

where $L$ represents the validation loss (e.g., cross-entropy), and $\alpha,\beta,\gamma,\delta$ are constants fitted from large-scale experiments. In this work, we not only reproduced similar scaling trends on vision tasks, but also validated the second component of the scaling law (the token count) in the vision domain. Specifically, we show that increasing token count in vision can be achieved by reducing patch size rather than solely increasing dataset size, which we believe is an important theoretical extension of the original scaling law foundation. We sincerely appreciate your suggestion on this point and will include the corresponding discussion in the revised version.

Q2: The benefit of smaller patches has been known and has been exploited for quite some time

Thanks. We agree that many existing studies have observed that using smaller patch sizes within a certain range can improve prediction performance. However, prior to our work, these observations remained scattered and lacked a unified theoretical or empirical framework. In contrast, our study elevates patchification scaling from isolated findings to a law-level conclusion.

We believe this distinction is substantial—not only in terms of the conclusions themselves, but also in their implications for guiding future progress in the community. For example, before the NLP scaling laws were formally introduced by Kaplan et al., practitioners had already noticed from experience that "larger models and more data tend to yield lower loss." Yet such heuristic insights were insufficient to offer reliable guidance or theoretical grounding for large-scale language model development.

The formulation of scaling laws provided the community with a much clearer signal: the scaling curve had no evident upper bound, or at least we were far from reaching it. This encouraged researchers to confidently invest significant resources in scaling up language models, ultimately contributing to the breakthrough success of models like the GPT series.

In the same spirit, we would like to highlight the contribution of our work. We aim to offer a long-term, principled perspective on vision model development. The value of patchification scaling laws lies in showing that patch size is a reliable scaling dimension—even at today’s typical input scales, shrinking the patch size down to 1×1 continues to yield noticeable gains. This suggests that for those seeking to push the limits of model performance, reducing patch size remains a viable and promising direction.

Q3: Concerning the argument that the scaling law can be attributed to the information loss in the patch creation step . Why should this hold in the case where the hidden dimensions are large enough to simply stack the corresponding pixels?

When the hidden dimension exceeds the total number of pixels within a patch, the patch embedding may theoretically support a lossless projection. However, in practice, the high-dimensional hidden features are optimized to represent the holistic semantic of the whole patch, rather than preserving the pixel-level representations. This is because the token mixing process (e.g., self-attention) is performed on patch-level features. That says, the different segments within a feature vector undergo the same operation in token mixing. Moreover, the hidden dimension cannot be flexibly scaled up since it quadratically increases parameter count for all linear projection layers and can easily lead to training collapse (e.g., also observed in Dehghani, et al., 2023).

Q4: Typo fig 3b.

Thanks. We will fix it in the revision.

We appreciate your comments and feedback during the review stage. We‘d like to respectfully point out that your review contains several factual misunderstandings of our paper. These points were clearly presented in the main text, and we'd like to first clarify them below:

1. Absence of 1x1 patch size for ViT: This is factually incorrect. We do implement pixel patchification in ViTs, albeit at lower input resolutions. The results are clearly shown in the first row of Tab.1.

2. “The overall method… showing results on vision and timeseries tasks.” Our work is solely focused on vision tasks, and there is no mention or evaluation on time series data anywhere in the paper.

3. “without ever indulging into the GLOPs, test and training time compute” This is completely inaccurate. We provide thorough analysis of FLOPs (Fig.3), memory usage and training time (Tab.6), and discuss computational costs of patch scaling in multiple places (e.g., Lines 324–329, 436l–408r).

4. "no ablation that discusses the patchification vs. embedding dimension size trade-off." We compared patch size and parameter size scaling in the first experiment of the ablation study in Section 4.3. The embedding dimension is highly equivalent to the parameter size. E.g., in the standard scaling of ViT models from ViT-T to ViT-B, the only change lies in the embedding dimension; from ViT-B to larger models such as ViT-L, it still involves increased embedding dim while depth is also changed.

Q1: Compare to 32 patch size

Our study focuses on exploring scaling laws of patchification, which means we need to examine both scaling up and scaling down. We aim to collect a wide-range performance curve to demonstrate that the gains are smooth and consistent across different patch sizes. The 32 patch size results are reported to provide a more comprehensive view; we did not hide the results for the mainstream 16 patch size—they are shown in the same table and scaling from it leads to a +2.2 box AP.

Q2: Generalization capability

We demonstrated that patch scaling consistently leads to improved predictive performance, which is generalizable for different models, tasks, and input resolutions. Below we include more results of generalization capability against out-of-domain data, where our conclusions hold true for ImageNet variant test samples as well.

Q3: Patchify in NLP

Word-level tokenization has indeed been prominent in NLP. However, we believe this is only a phase-based conclusion but no one has ever proven that it is the only correct approach. In contrast, recent studies(e.g. Byte Latent Transformer) have shown that byte-level LLMs can outperform word-level models and offer better scaling potential. Thus, whether patchify is necessary remains an open question that deserves further investigation. Moreover, directly transferring conclusions from NLP to CV is not always appropriate. Images and text differ significantly in terms of information density; a pixel cannot be naively equated to a character in text. These differences call for careful study, and our work aims to contribute to this ongoing exploration.

Q4: Quantify the information loss from the information theory perspective.

This paper is not a theory-oriented study, but rather an empirical investigation into scaling laws. We believe both empirical and theoretical perspectives are equally important, and our experimental results can serve as a solid foundation for future theoretical work. In fact, our evaluations already offer explanations from an information point of view, in which compression can be assessed via entropy differences. Fig.1 explicitly shows how cross-entropy loss changes during patch scaling, which is actually measuring the KL divergence between ground-truth distribution and encoded distribution. This typically serves as a direct proxy for compression in information theories.

Q5: Results marginally better; underperform SOTA models

Our focus and contribution is figuring out how patch size affects the performance of vision models, rather than chasing SOTA on open-ended architectures. Whether a performance gain is considered marginal is subjective and varies by context. For fixed architecture, improving accuracy from 82.6 to 84.6 is a significant achievement. It is widely acknowledged that hierarchical models (e.g., Swin, SpatialMamba) are more capable of fitting ImageNet-scale datasets compared to plain models, but plain models have wider applications in multimodal tasks. We focus on patch scaling within plain architectures so the performance gap with SOTA models is expected. In our response to R#DwN7 we also show that patchification scaling laws hold true for Swin, so we believe that future follow-up work applying patch size scaling to SOTA architectures could reasonably expect further performance improvements.

We appreciate the reviewer’s constructive feedback and detailed suggestions. The detailed response to your questions/concerns are presented below:

Q1: Additional experiments and discussion about sparse and spatially local attention networks.

A1: Thank you for your constructive suggestion. We followed your feedback and conducted additional experiments on Swin Transformer, with the summarized results provided below. Specifically, we use Swin-T as the base model, which has an initial patch size of 4×4 and hierarchically downsamples the feature map by a final factor of 8 in both width and height during intermediate stages. We scaled it up by reducing the initial patch size to 2×2 and 1×1, while keeping the default downsampling rates unchanged. This scaling strategy closely mirrors the approach we used when handling standard ViT models. As shown in the table, our conclusion also holds for Swin Transformer—smaller patch sizes consistently lead to lower test loss and improved accuracy. In the revision, we will include larger local-attention models such as Swin-S and Swin-B to further support our findings.

Q2: Limitations of input size scaling.

A2: We appreciate your insightful comments on this matter. We would like to explain the motivation behind this set of experiments: in fact, input size scaling and patch size scaling are largely equivalent—they both proportionally change the feature processing granularity and the sequence length of the model. Through these experiments, our goal is simply to show that patch size is a more suitable choice for a standardized direction of scaling, since it does not affect input storage, and input size scaling does not necessarily offer performance advantages over patch size scaling beyond a certain range.

Specifically, we know that increasing the input size within a certain range tends to improve model performance. For example, on ImageNet, using the same model such as DeiT-Base/Patch16, an input resolution of 384×384 achieves noticeably better accuracy compared to 224×224 (e.g., 83.1 vs. 81.8). We believe this improvement basically comes from two sources: 1) the direct benefit of increased computation from scaling up the input; and 2) a higher input resolution reduces the distortion caused by resizing images during preprocessing. To ablate these two effects, we keep a fixed ratio between input size and patch size, which helps maintain a roughly constant computational cost and figure out the impact of the second term. Then our experiment demonstrates that this benefit tends to vanish once the input resolution exceeds the average original image size in the dataset.

We will carefully rephrase the description and analysis of this experiment in the revised version based on your suggestions and comments.