Weakness 1. The comparisons lack fairness. Only if the superior performance is maintained under these fairer conditions can the effectiveness of the proposed method be truly demonstrated.

To address the reviewer’s concerns about comparative fairness, we designed controlled experiments by constraining UniViTAR’s ​​per-image token range​​ to align with the computational budgets of baseline encoders (SigLIP-SO400m: 729 tokens; AIMv2-H: 1024 tokens). Specifically, we enforced a token range of ​​288-1170​​ (mean≈729) and ​288-1760​​ (mean≈1,024) during training/inference. Results on three high-resolution benchmarks in below table demonstrate UniViTAR’s consistent superiority under matched computational conditions:

Weakness 2. The paper lacks intuitive case studies and theoretical analysis. Does the native resolution method perform better in all scenarios, or only in cases of extreme aspect ratios or resolutions?

We sincerely appreciate the reviewer's insightful comment regarding the need for deeper theoretical analysis and intuitive case studies.

• Theoretical advantages of native resolution processing​​.
  • ​​Preservation of Geometric Integrity.​ Fixed-resolution methods inevitably introduce geometric distortion, either through peripheral information loss (center cropping) or object deformation (aspect ratio warping). In contrast, native resolution processing inherently eliminates such distortions by: (1) maintaining the original aspect ratio without spatial modification, and (2) preserving fine-grained contextual relationships across the entire image canvas.
  • ​​Alignment with Natural Image Resolution Distribution​.​ Native resolution processing better reflects the inherent variability in resolution found in real-world images.
• The native-resolution approach demonstrates superior performance or greater flexibility in most scenarios. However, it is not universally optimal. Below, we provide some intuitive case studies to delineate specific scenarios:
  • ​​Hardware Compatibility Constraints​​. Modern GPU tensor cores optimize for fixed-dimension computations. Variable-length token sequences cause suboptimal hardware utilization.
  • ​​Extreme Aspect Ratio Scenarios (>1:100)​​. For extreme OOD-resolution inputs, both fixed and native-resolution methods empirically fail to extract good visual features due to inherent limitations in training data coverage.

Question 1. Does the training scheme based on native resolution present efficiency issues? Compared to directly maintaining the image’s native resolution, this approach significantly increases computational resource requirements during training.

We appreciate the reviewer's insightful questions regarding the computational efficiency of our native-resolution training scheme and the comparison to approaches like Qwen2-VL. We address these points below:

• We contend that directly training on native resolution (analogous to Qwen2-VL) presents several disadvantages, making it a suboptimal solution:
  • ​​Requirement for pre-packing​.​ To maximize data utilization efficiency and training throughput, a ​​greedy pre-packing strategy​​ is typically adopted (as utilized in NaViT). Critically, different packing strategies can ​compromise data randomness​​ during training.
  • ​​Constrained resolution distribution​.​ The ​​resolution distribution​ becomes rigidly determined solely by the training dataset, which leads to a ​​reduction in the diversity of resolutions​ encountered during training. Our proposed strategy dynamically adjusts the token sequence length per image ​within each batch​​, based on ​​both​​ the distribution of image resolutions within the current batch and the predefined ​maximum token limit​. Consequently, the same image may be resized to different resolutions across distinct training batches, and this inherent variability ​​ensures a richer distribution of resolutions.
• We argue that our proposed strategy ​​does not impose a significant increase in computational resource demands​​. The primary reason is as follows. During training with batch size greater than 1, training frameworks invariably ​​pad variable-length visual sequences within a batch to a maximum sequence length​​ for computational efficiency and minimize ​pipeline bubble. Crucially, regardless of Batch-Padding-to-Max-Length strategy in Qwen2-VL or Batch-Resizing-to-Max-Length strategy in UniViTAR, the ​upper bound​ on computational resource requirements is ​more dictated by the configured maximum sequence length. In fact, by resizing images within each batch according to our strategy, our method actually ​​reduces the overall padding ratio​ across the training process and ​enhances computational resource utilization efficiency​​.

Question 2. Does scaling images to the maximum sequence length during training, while keeping their native resolution during inference, impact inference performance?

• We first wish that we have correctly interpreted the reviewer’s inquiry. The proposed native-resolution training strategy does not uniformly scale images to a maximum sequence length. Instead, ​​the resolution of each image is dynamically adjusted based on the resolution distribution of all images within its batch​​. Under this paradigm, the same image may be resized to different resolution when processed in distinct batches, which may be ​​upscaled​​, ​​downscaled​​, or ​​remain unchanged​​ depending on the resolutions of other images in the same batch.
• Regarding the visualization results for Figure 3 (right panel), under strict visual sequence length constraints (256 tokens), we think the suboptimal performance of Native Aspect Ratio primarily stems from UniViTAR’s early training stages bias (Stage-1/2), where a fixed sequence length of 256 tokens was employed for pretraining. Crucially, as illustrated in the Figure 3, when maximum visual sequence lengths exceed 256 tokens (e.g., ≥576), the following relationship consistently holds: Native Aspect Ratio​​ > Fix-size.
• Exploring the impact of minimum resolution on benchmark performance is indeed a valuable suggestion. We adjust the min-token number during evaluation, and the results are shown in the table below. Interestingly, performance improvements on certain benchmarks are not linear, which aligns with the findings in Qwen2-VL. Benchmarks like OCRBench contain many very small images, and excessive enlargement of these images leads to distortion of images. Overall, the average performance across the 16 benchmarks shows little difference, indicating robustness of our model. | Min Tokens | Avg. Performance|HallusionBench|OCRBench| |:---:|:---:|:---:|:---:| |72 |49.3|57.4|37.0| |144|49.5|57.4|36.8| |288|49.5|57.8|36.9| |576|49.5|57.6|36.7|

Question 3. How efficient is inference with ViT using native resolution? When the image resolution is very high, a large number of tokens will inevitably be generated.

The integration of native-resolution processing in ViT inherently establishes a fundamental efficiency-performance trade-off, governed by the quadratic computational complexity of self-attention mechanisms relative to token count. This implies that inference efficiency degrades progressively as token volume increases—a theoretical limitation scaling polynomially with sequence length. To mitigate computational explosion during dynamic-resolution training, we impose constraints on ​​maximum tokens per image​​, i.e., CLIP pretraining phase​ is 16,384 tokens and MLLM alignment phase​ is 5,832 tokens (after pixel-unshuffle is 2916). This calibrated token ceiling balances inference efficiency against representational fidelity, ensuring tractable resource utilization while preserving performance gains.

Question 4. Did the authors analyze how the distribution of image resolutions in the training data affects the model’s performance on tasks requiring varying input sizes?

Our analysis reveals that native-resolution vision encoder exhibits non-trivial ​​extrapolation capabilities​​ to resolutions beyond its training distribution. To quantitatively assess this, we conducted an experiment using the ​UniViTAR-0.6B + Qwen2.5VL-1.5B​ as baseline​. During training, the maximum token count was limited to ​2,916 tokens​​ (corresponding to visual tokens number fed into the LLM). During inference, we progressively increased the evaluation maximum token limit per image to ​5,832 (2×)​​ and ​8,748 (3×)​​ tokens. Performance across our full evaluation suite and specialized high-resolution benchmarks is summarized below:

Increasing the maximum token limit to 5,832 (2×) and 8,748 (3×) yields ​​consistent improvements​ on benchmarks containing substantial high-resolution content, namely ​HRBench4K​ and ​TextVQA_VAL. Furthermore, this result complements the low-resolution extrapolation capability demonstrated in the Question 2, ​​collectively confirming​ the model's inherent robustness across diverse resolutions beyond its specific training distribution.

Weakness 1. One limitation of the paper is the lack of discussion on related work that tackles similar problems using different architectures. For example, causal convolutions, which are widely used in video generation, have also been applied in encoding tasks for both images and videos, and naturally support variable-length or multi-resolution inputs. Similarly, diffusion transformer (DiT) models for image and video generation share many architectural elements with UniViTAR, such as Transformer backbones with RoPE, QK-Norm, and progressive resolution handling. While these models target generative tasks, they address similar structural and resolution challenges.

We sincerely thank the reviewer for this insightful and valuable comment regarding the scope of the related work discussion. Expanding our discussion to encompass these approaches would have significantly strengthened the context of our work.

• ​​Discussion of Causal Convolutions​.​ We agree that architectures employing ​causal convolution​ represent an important alternative strategy for handling sequential data, including images and videos, particularly due to their inherent support for ​variable-length inputs​​ and multi-resolution representations. While our work primarily focuses on the quality of visual semantic-level representations and their generalization capabilities in multimodal understanding scenarios, adopting the alternative approach of causal convolutions would introduce multiple substantial limitations:

  • Loss of Global Context Modeling. Bidirectional attention in UniViTAR (transformer-centric paradigm) dynamically computes relationships between all visual tokens, enabling holistic understanding of scene context and object interactions. Causal convolutions' locality fundamentally restricts context aggregation to receptive fields defined by kernel size/depth.
  • Suboptimal Static Image Processing. For non-sequential visual inputs (static images), causal convolutions impose an artificial, task-irrelevant directional bias that has no perceptual justification (​e.g.​, prioritizing left patches over right). This conflicts with the isotropy of natural images.
  • Representational Capacity Constraints. While deep stacks of dilated convolutions can increase receptive fields, they remain fundamentally less expressive than attention mechanisms for capturing complex, long-range token dependencies. This may necessitate significantly deeper architectures to compensate, diminishing potential efficiency gains.

• ​​Discussion of Diffusion Transformers (DiTs):​ ​While DiTs and UniViTAR share some architectural elements, we note DiT's primary application domain is ​generative modeling​, whereas UniViTAR is primarily designed for ​discriminative representation learning​. This key distinction in objective leads to substantial differences in training regimes, loss functions, and final application. Moreover, it is crucial to emphasize that the core contribution of UniViTAR lies in its complete training scheme for native resolution-based Vision Transformers in the open-source community and a family of homogeneous visual foundation models for multimodal undersanding research.

Weakness 2. While positioning UniViTAR as a vision encoder for MLLMs is arguably the most impactful application of this work, the comparison is limited and outdated. The paper mainly contrasts with earlier vision encoders, but omits recent models that specifically support native resolution, such as NaViT and FlexiViT. Furthermore, several recent works focused on multimodal understanding—like Scaling Language-Free Visual Representation Learning and Cambrian-1—propose advanced vision encoders tailored for MLLMs.

We sincerely appreciate the reviewer’s valuable feedback. Below, we provide comparative analyses of additional relevant visual foundation models referenced in your comments.

• NaViT​[1]. By introducing a token-packing strategy, NaViT pioneered the application of native-resolution training schemes in conventional Vision Transformers, concurrently exploring efficiency improvements in native-resolution processing. Detailed distinctions and advantages of UniViTAR relative to NaViT are comprehensively addressed in our response to Reviewer dNd7. To minimize redundancy, we refrain from reiterating these points here and kindly refer the reviewer to that discussion. For quantitative validation, we present performance comparisons between UniViTAR and NaViT on linear-probing classification, demonstrating UniViTAR-0.3B’s significant advantages.

• ​FlexiViT​​[2]. This work is referenced in the introduction section of our main text. FlexiViT dynamically adjusts patch sizes to generate variable-length visual token sequences, accommodating both fixed and native-resolution inputs. We acknowledge its exclusion from our primary comparative analysis and clarify two key reasons:

  • ​​Model Scale​​: FlexiViT’s published results predominantly utilize ViT-B architectures, while UniViTAR’s smallest variant (UniViTAR-L) operates at approximately 300 million parameters, rendering direct performance comparisons inequitable.
  • ​Training Paradigm​​: FlexiViT-L employs supervised pretraining on ImageNet, whereas UniViTAR adopts large-scale contrastive pretraining without ImageNet data. This fundamental methodological discrepancy precludes rigorous equivalence in comparative evaluation.
  • Nevertheless, we still provide supplemental comparisons in the following table: one is the comparison results of linear-probing on ImageNet variants (Table 1), and the other is the results of the model on the zero-shot classification and cross-modal retrieval when FlexiCLIP-B at patch-size 8 (Table 2).

• Web-SSL​[3]. Web-DINO explores visual self-supervised learning paradigms, demonstrating that scaling both data volume and model size beyond the billion-scale can yield performance approaching that of CLIP-trained vision foundation models. However, a non-trivial performance gap persists between Web-DINO and CLIP-paradigm representations, as evidenced by model parameter disparities (7B vs. 1.4B) and benchmark comparisons. We provide preliminary performance comparisons in the following Table.

• Regarding multimodal capabilities, we excerpt key results from Table 3 of Web-SSL[3] into the following Table for the reviewer’s reference. Notably, Web-DINO at 7B scale underperforms SigLIP-SO400M-378 in multimodal tasks. Similar observations were documented in Cambrian-1[4]. Given the necessity for rigorous multimodal benchmarking, we will incorporate expanded comparative analyses in the revised manuscript.

• Cambrian-1[4]. Cambrian-1 introduces a family of multimodal LLMs designed with a vision-centric approach, which mainly employs the MoF (Mixture-of-Feature) paradigm for vision representation with a combination of four exist vision encoders: OpenAI CLIP ViT-L/14@336, SigLIP ViT-SO400M/14@384, OpenCLIP ConvNeXt-XXL@1024, and DINOv2 ViT-L/14@518. Considering that UniViTAR primarily focuses on (1) exploring large-scale pre-training strategies for a ​​single visual encoder​​ under ​native-resolution setting​ and (2) improving the quality of ​​general visual representations​​, we have provided detailed comparisons in the paper between UniViTAR and the CLIP series models employed in Cambrian-1 (see Experiments 3 to 6 in our paper). Generally, the MoF fusion scheme reduces information loss by integrating features from multiple visual representations. However, this approach typically incurs ​​increased computational overhead​​ and introduces complexities in selecting and integrating diverse ​​visual foundation models​​ for practical applications.

To ensure ​​comprehensive comparison​, we will incorporate detailed discussions of these points in the ​updated version​​ of our manuscript.

[1] Dehghani, Mostafa, et al. "Patch n’pack: Navit, a vision transformer for any aspect ratio and resolution." Advances in Neural Information Processing Systems 36 (2023): 2252-2274.

[2] Beyer, Lucas, et al. "Flexivit: One model for all patch sizes." ​Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition​. 2023.

[3] Fan, David, et al. "Scaling language-free visual representation learning." arXiv preprint arXiv:2504.01017 (2025).

[4] Tong, Peter, et al. "Cambrian-1: A fully open, vision-centric exploration of multimodal llms." Advances in Neural Information Processing Systems 37 (2024): 87310-87356.

Weakness 1. The core architectural design and training strategy closely follow established approaches in the community, especially Qwen2VL and NaViT. The work mainly consolidates community best practices rather than proposing new model components or learning algorithms.

Question 1. What concrete differences (in architecture, training, or results) set UniViTAR apart from Qwen2VL's vision encoder? A table would help clarify any unique contributions.

We sincerely appreciate the reviewer’s perspective regarding UniViTAR’s consolidation of community best practices and value their constructive feedback. Pioneering works such as NaViT introduced native-resolution training schemes for Vision Transformers (ViTs), exploring efficiency improvements in native-resolution processing. Similarly, Qwen2VL’s vision encoder adopted NaViT’s principles by leveraging pretrained fixed-resolution visual foundation model (​e.g.​, DFN[1]) to advance multimodal representation alignment and pretraining for large multimodal models. In our response to ​​Reviewer dNd7​​, we comprehensively contrasted UniViTAR’s distinctions and advantages relative to NaViT and Qwen2.5-VL’s vision encoder. To avoid redundancy, we kindly invite you to refer to the relevant sections of our reply to Reviewer dNd7 for more detailed comparative analyses. Below, we explicitly delineate key differences between the UniViTAR framework and Qwen2-VL’s vision encoder:

Question 2. How much benefit can be gained by using Dinov2 for distillation in Stage 1?

Thank you for your perceptive and highly professional inquiry. We did conduct a thorough investigation into the performance gains offered by the distillation branch during the initial model exploration phase. The following summarizes our empirical observations and experimental findings:

• ​Acceleration of early-stage convergence.​ Our experiments on UniViTAR-0.3B involved training on 3B image-text samples from scratch. Evaluating checkpoints at different training intervals (every 0.5B samples) within the same training cycle revealed that utilizing Dinov2 for distillation significantly enhances convergence during the early training phase. For instance, at 0.1B samples, distillation supervision yielded a performance gain of 17.3 points on the ImageNet-1K zero-shot classification. This beneficial effect diminishes progressively as training advances, as detailed in the subsequent comparison.

• ​​Improvement in final model performance.​ Evaluation of the final model trained on 3B image-text pairs demonstrated that employing Dinov2 as the distillation branch consistently delivers an average performance gain of 2.1 points across six zero-shot classification benchmarks. The detailed comparative analysis is presented below.

Question 3. On vision-language benchmarks, the performance gain of scaling the vision encoder from 0.6B to 1B seems to be limited. What is the author's perspective on this phenomenon ?

We sincerely appreciate this insightful observation regarding the scaling behavior of vision encoders in vision-language models. As the capacity of the visual encoder increases, we observe that the VLM model's performance on established vision-language benchmarks becomes increasingly constrained by factors other than pure visual representation power. We would like to humbly submit the following perspectives on this phenomenon:

• ​​Cross-modal Alignment Limits.​ Larger vision encoders inherently possess greater potential for highly specialized visual feature extraction. However, the capacity and optimization of the modality fusion mechanism may become a bottleneck, limiting the utility of richer visual features.
• ​Late-Fusion[2] Architecture Bound.​ Current late-fusion methodologies typically entail integrating independently pre-trained modality-specific models, such as connecting vision encoders to LLMs and continuing multimodal training. Crucially, each constituent module (e.g., vision encoder, LLM) exhibits distinct hyperparameters, distinct pre-training data mixtures, and distinct scaling properties with respect to computational resources and training data volume. Exploring the joint scaling strategy of visual encoders and LLMs may be the most necessary point in the future work.
• ​​Limited Train Data and Benchmark Saturation.​ Our models in Table 6 were trained on a comparatively limited multimodal dataset, i.e. research-level, consisting of only 2.5M image-caption pairs and 665K instruction tuning samples. Furthermore, the visual encoder parameters remained frozen throughout training for more accurately verifing the ability of pre-trained visual representation. This configuration resulted in diminishing performance gains relative to increasing visual encoder capacity. On the other hand, some benchmarks may lack the granularity or difficulty to fully reflect the nuanced improvements gained from larger vision encoders, leading to apparent saturation.

[1]. Fang, Alex, et al. "Data filtering networks." arXiv preprint arXiv:2309.17425 (2023).

[2]. Shukor, Mustafa, et al. "Scaling laws for native multimodal models." arXiv preprint arXiv:2504.07951 (2025).

Weakness 1. While the paper emphasizes native resolutions, existing models like Qwen2.5-VL already adopt native resolution strategies, reducing the novelty of the contribution.

Question 1. How does UniViTAR differentiate itself from prior native-resolution approaches like NaViT and Qwen2.5-VL?

We sincerely appreciate the reviewer’s insightful perspective regarding native-resolution approaches and their valid inquiry into UniViTAR’s differentiation. We consider Qwen2.5-VL and NaViT to be significant as previous implementations of native-resolution processing. However, UniViTAR introduces ​several core innovations​​ that fundamentally differentiate it from these prior efforts in terms of both technology and functionality.

Comparison with NaViT​​. As an early exploratory work, NaViT's core contribution lies in introducing a native-resolution token packing training strategy to Vision Transformers (ViTs), thereby pioneering efficiency improvements in native-resolution training. While its insights indeed provide foundational inspiration for contemporary native-resolution models, UniViTAR demonstrates distinct advantages:

• ​​Scalability and Homogeneity​​. UniViTAR offers larger and more diverse model scales (0.3B/0.6B/1.4B), with all models sharing structural and training homogeneity across sizes and data sources. This facilitates granular community research into scaling both vision foundation model and multimodal large language model.
• ​​Flexible Native-Resolution Training Strategy​​. NaViT primarily employs greedy token packing with sequence padding to fixed lengths during training. In contrast, UniViTAR adopts a random packing strategy combined with batch-wise sequence resizing (#L149~#L153).
• ​​More Visual Modality​​. UniViTAR employs a multi-stage training paradigm (image pretraining → image-video joint training), achieving state-of-the-art performance on both image and video benchmarks. NaViT, limited to image-only pretraining, underperforms many CLIP-based counterparts.
• ​​Advanced Architectural Components​​. For transformer-centric native-resolution modeling, adaptive positional encoding is essential: while NaViT uses factorized positional embeddings, UniViTAR implements 2D-RoPE to encode visual sequences with variable lengths, critically enhancing positional awareness.

Comparison with Qwen2.5-VL-ViT​​. As a leading open-source Vision-Language Model (VLM), Qwen2.5-VL similarly adopts a native-resolution multimodal training strategy akin to NaViT. UniViTAR demonstrates the following distinctive advantages:

• ​Purer Native Dynamic-Resolution Representation​​. Qwen2.5-VL’s vision encoder undergoes extensive multimodal training (primarily on in-house data) across multiple multimodal stages (multimodal pretraining → post-training), resulting in representations highly adapted with Qwen-series LLMs. This alignment substantially constrains ​generic visual transferability​. UniViTAR, conversely, employs contrastive learning only for native-resolution pretraining, delivering superior generic visual representations.
• ​Granular Training Recipe and Model Scalability​​. While Qwen2.5-VL releases pretrained VLM weights, it withholds the visual pretraining weights in CLIP phase and critical implementation specifics. UniViTAR systematically documents training methodologies and releases multi-scale models (0.3B/0.6B/1.4B VS. 0.6B), establishing a comprehensive research foundation for the community.
• ​​Architectural Distinction in Attention Mechanism​​. UniViTAR utilizes ​​Full-attention​ exclusively, whereas Qwen2.5-VL employs a hybrid ​​Full-attention and Window-attention​ design. Although window-attention improves computational efficiency, it typically incurs non-trivial performance degradation in visual representation quality. Considering the versatility of downstream tasks and the fact that the scale of language models is usually much larger than that of visual encoders, UniViTAR uses Full-Attention to preserve the visual representation capabilities as much as possible. However, improving the training and inference efficiency of visual encoder at native-resolution will be our next important work.

Weakness 2. The multimodal performance (Table 6) is relatively modest, and extensions to large-scale MLLMs remain under-explored.​

We sincerely appreciate the reviewer’s insightful observation regarding the multimodal performance in Table 6 and the need for large-scale MLLM exploration. Due to the research-level multimodal traning data, we acknowledge that UniViTAR’s current multimodal results are not state-of-the-art, as our primary focus was rigorously validating the ​​generic visual representation quality​​ of dynamically scaled vision encoders and other state-of-the-art vision encoders—rather than optimizing end-to-end MLLM systems.

To address reviewer's concren, we have conducted new experiments scaling UniViTAR to larger MLLM frameworks, e.g., integrating UniViTAR-0.6B with Qwen2.5-1.5B/3B/7B. Results demonstrate clear scaling laws in the table below, which confirms UniViTAR’s scalability potential when paired with larger language models.

Exploring UniViTAR's generalization capabilities on larger language models (​*e.g.*​, 70B parameters) and expanded data scales constitutes our critical next work to further validate the versatility of the UniViTAR family. We thank the reviewer for highlighting this gap—it directly shapes a more impactful contribution.

Weakness 3. The paper would benefit from more granular ablation studies on the architectural choices (e.g., LayerScale, RMSNorm) and their impact across both classification and multimodal tasks.

We sincerely thank the reviewer for the constructive feedback regarding the need for more granular ablation studies. We acknowledge that deeper analysis of architectural components is critical for understanding UniViTAR’s design efficacy. To address this, we established a validation baseline based on UniViTAR-0.3B, with the overall training size controlled at 0.5B scale (Merge1B for 0.5 epoch). We validated the effectiveness of the three modules, LayerScale, RMSNorm, and SwiGLU. The specific results are as follows:

The ablation studies summarized in Table yield the following conclusions:

• All three investigated modules—SwiGLU, RMSNorm, and LayerScale—enhance visual representation capabilities, albeit to varying degrees. SwiGLU yields the most substantial gains (+1.75%), followed by RMSNorm (+0.75%), and LayerScale (+0.33%). Their synergistic integration delivers cumulative performance improvements (+2.34%), confirming complementary benefits.
• SwiGLU and RMSNorm, widely adopted in large language models (e.g., LLaMA), demonstrate comparable efficacy in visual representation learning. This validates their generalizability as universal architectural components across modalities.

These ablation results and associated discussions will be incorporated into the revised manuscript to clarify the impact of individual design choices.