SmolVLM: Redefining small and efficient multimodal models

We sincerely thank reviewer R-BnL7 for their detailed and comprehensive feedback. We are pleased with the thorough review full of actionable points. We are happy that the reviewer finds our insights useful for the community.

1. While we could not control for encoder resolution, with image splitting, all experiments had comparable Indeed, vision encoders with larger input resolutions have greater capacity, but at that time, there wasn't a Siglip So 400M with an input resolution larger than 384, so we couldn't make a comparison with the same input resolution. We also didn't want to limit our small vision encoder by using one with a lower resolution. However, since we are employing image splitting, the effective overall image resolution during training is comparable. During evaluation, images are rarely small enough that they wouldn't be split, so both models handle them at similar resolutions.
2. We will modify Figure 3 (left) to use curves corresponding to fixed LM sizes for readability. Thank you for the suggestion.
3. We will improve Figure 3 (right) for readability. The different colors are not necessary – there is no difference.
4. In Figure 3, the performance difference between the 256M model at r=2 and r=4 falls within the noise of the benchmarks, making it difficult to determine which is superior. However, for the 500M model, it is evident that r=4 performs better. Notably, r=4 significantly reduces the number of tokens required for both models, leading to a much more efficient model. Therefore, we assert that r=4 is clearly beneficial for both model sizes. We will clarify that when we say "small models benefit from r=4," we mean that although the quality may not necessarily improve, the quality/performance trade-off is quite favorable.
5. This is because the difference between the 1.5 and 2.5 average durations was much larger than the difference between the 2.5 and 3.5 average durations. We will clarify that the intention is to start saturating at 3 minutes, not 3.5 minutes.
6. We will add the following table to our manuscript:

Along with an experimental section detailing additional training details.

Answer to Questions: Term positional tokens: When we split images to pass them to the LLM, we added special tokens to the text tokenizer, which are later learned during SFT. This is the effect of concatenating a learnable embedding before each image patch, allowing the model to more easily identify it. Thank you for pointing out that this wasn't very clear in the original manuscript; we will revise it to enhance clarity.

We sincerely thank Reviewer jgcB for their evaluation and valuable feedback. We are pleased the reviewer recognizes the high bar we set with the evaluations we chose, the thorough layout we set for design choices in reducing the size of multimodal models, and our commitment to open-source contributions, including the mobile application.

Below we address the key points raised:

1. Evaluation metric insufficiency:

We agree that VRAM alone does not fully validate real-time performance. Thus, we will add additional figures with comprehensive wall-clock throughput metrics (Tokens Per Second, Time-To-First-Token) measured on both consumer hardware (MacBook M1-M4 Max series) and server-side GPUs (H100s). For example, SmolVLM-256M achieves up to 45 tokens/sec on a three-year-old MacBook Air M2 with 16GB of RAM, demonstrating practical real-time capabilities. More information on this can be found in the general rebuttal.

2. Small VLM direct comparison:

Regarding energy efficiency, direct comparisons to other models across diverse hardware remain challenging; however, our results on the H100 GPUs indicate SmolVLM variants consistently outperform comparable models in tokens processed per second by at least 3 times, strongly suggesting superior energy efficiency per token.

We thank R-RaaY for their detailed and comprehensive review. We are pleased the reviewer recognizes the significance of targeting edge device limitations, appreciates our systematic design-space exploration, comprehensive benchmarking, and our commitment to open-source contributions, including the mobile application.

Below we address the key points raised:

1. Questionable comparison target in the abstract

We only compare to Idefics-80B to motivate the exploration of small VLMs, and illustrate how much the field has progressed, as a 256M model outperforms an 80B model from 20 months ago. We will revise the abstract and include comparisons to other small, newer VLM models (e.g., InternVL2-1b/LLaVA-OV-0.8B). Figure 1 (right below the abstract) contains more relevant comparisons.

2. Primarily engineering-oriented, with limited novelty

Please note that SmolVLM is the first VLM with fewer than 800M parameters (256M and 500M). This small size allows it to perform inference on devices with very limited resources, and it also enables exceptionally high throughput at large batch sizes.

3. Incomplete and potentially biased evaluation.

(A) Edge benchmarking

We agree that VRAM alone does not fully validate real-time performance. Thus, we will add comprehensive wall-clock throughput metrics (Tokens Per Second, Time-To-First-Token) measured on both consumer hardware (MacBook M1-M4 Max series) and huge GPUs (H100s). For example, SmolVLM-256M achieves up to 45 tokens/sec on a three-year-old MacBook Air M2 with 16GB of RAM, demonstrating practical real-time capabilities. More information on this can be found in the general rebuttal.

(B) Comparisons to other models in the experimental section.

On the paper's first page, we compared SmolVLM to several small VLMs based on average performances. Due to the page limit and the number of experiments shown in the manuscript, we only compared selected models for individual benchmarks. However, to enhance reader convenience, we will include the comprehensive table below, including MiniCPM-V and NVILA.

Image-only benchmarks

Video benchmarks

We sincerely thank R-ofar for their positive evaluation, confidence in our work, and valuable feedback. We are pleased they recognize the significance of targeting edge devices and appreciate our systematic design-space exploration, comprehensive benchmarking, and commitment to open-source contributions, including the mobile application.

Below, we address the points raised in their detailed review:

1. Real-time Claims: Wall-clock Throughput and Energy Metrics

We agree that VRAM alone does not fully validate real-time performance. Thus, we will add additional figures with comprehensive wall-clock throughput metrics (Tokens Per Second, Time-To-First-Token) measured on both consumer hardware (MacBook M1-M4 Max series) and server-side GPUs (H100s). For example, SmolVLM-256M achieves up to 45 tokens/sec on a three-year-old MacBook Air M2 with 16GB of RAM, demonstrating practical real-time capabilities. More information on this can be found in the general rebuttal.

Regarding energy efficiency, direct comparisons to other models across diverse hardware remain challenging; however, our results on the H100 GPUs indicate SmolVLM variants consistently outperform comparable models in tokens processed per second by at least 3 times, strongly suggesting superior energy efficiency per token.

2. Spatial Detail Loss with High Pixel Shuffle Ratios (r=4)

We thank the reviewer for their valid concern regarding spatial detail loss with high shuffle ratios. Our adoption of r=4 for small models (reducing 512x512 patches from 1024 to 64 tokens) aimed to balance computational efficiency with general-purpose performance.

Preliminary experiments at r=8 (16 tokens) showed significantly degraded performance after 24 hours on 64 H100 GPUs, making full training impractical and resource-prohibitive. Thus, r=4 was the most aggressive feasible compression. While a comprehensive error analysis on localization-heavy tasks for r=4 was not conducted, our reported OCR benchmarks are promising.

In the revised manuscript, we will concisely discuss the trade-off of the r-parameter, our rationale for r=4, the limitations of our experiments (including the absence of the specific error analysis), and highlight detailed evaluation of shuffle ratios on localization-sensitive tasks as important future work.

3. Ablations for Alternative Compression Methods (e.g., Patch-merging, Adaptive Pooling)

We acknowledge the reviewer's suggestion regarding ablations for alternative compression methods. We initially explored some options before deciding for pixel-shuffle as it is a straightforward yet highly effective compression method. Given our extensive existing analysis, exploring multiple additional compression techniques would significantly expand the scope. Tronchon et al. (Building and better understanding vision-language models: insights and future directions) explored some of these compression methods.

In the revision, we will clearly state that comparative analysis of alternative compression methods is an exciting future research direction. Our current results provide a robust baseline for understanding their effectiveness.

We appreciate R-ofar's insightful comments and strong support. We believe these clarifications and added analyses considerably strengthen our manuscript.

Dear R-ofar,

We conducted additional experiments to address the concerns raised by other reviewers regarding evaluation (measurement of TTPS and TPS, full table comparing more models in our response to R-RaaY).

Do these not meet your concern, and if not, what additional comparisons/experiments would you need to see to address this concern?

Best, The Authors