Quantization

Quantization is indeed relevant for model compression. While the focus of Little-Big is on inference-time compute savings without retraining, we acknowledge that quantization could further complement Little-Big. We have aded a brief discussion in Section 2.3, positioning quantization as a complementary method that can be integrated with Little-Big for additional gains.

Additional Experiments We also want to request the reviewer to please check our summary response that reports our findings in extending Little-Big to novel tasks and domains including semantic segmentation, zero-shot classificaiton, multi-label classifcaition as well as extensions to text classification tasks. Details of these new added experiments can be found in Appendix A.1 of the revised paper.

Fair comparison with Baselines

Thank you for this question. In the A-ViT paper by Yin et al., the authors report the baseline DeiT performance as 78.9%, whereas the original DeiT paper reports a performance of 79.81% for the same model. Additionally, we confirmed the 79.81% performance through the publicly released DeiT-S checkpoint. Since it was not immediately clear which of these values to report, and to ensure fairness when comparing against other methods, we initially included both. However, we acknowledge that this choice could cause confusion for the reader. To address this, we have revised the paper to only include the 79.81% baseline from the original DeiT paper for clarity and consistency.

We also made similar changes in Table-3 with DeiT-B architecture to improve readability. Little-Big pairs with T=0.0 indeed suggest that a big model can be replaced by a small model. In the original table, we included such pairs mainly to highlight that an effective but often neglected way of model compression is to train a smaller model with a better training recipe (e.g. DeiT3-S outperforms DeiT-B). But we acknowledge that it may cause confusion and removed such pairs with T=0.0 from Table 3.

Choice of Little Network

Thank you for pointing this out. We want to highlight that the choice of the "Little" network is, in fact, user-dependent and flexible. Little-Big imposes no strict requirement on which model to use as the Little model, and it is entirely architecture-agnostic. Users can select a Little model based on their specific constraints, such as available compute resources, latency requirements, or model compatibility with the Big model.

Typographical Error

Additionally, the phrasing “confidence > 0.5-0.7” was indeed an error and instead It should have been confidence > 0.5. To clarify, our intent was to convey that confidence in the models we examined inversely correlates with hardness. Specifically, from the distribution of correctable mistakes, we observed that most of these samples exhibit lower confidence. Thus, this observation directly motivates our approach of thresholding over confidence: when the confidence of a prediction falls below the threshold, the Big model is invoked to handle the sample. We thank you for highlighting this issue, and we have updated the phrasing and explanation in the revised paper for clarity and accuracy.

Additional References We have added the following references to the line "human vision ingests gigabits/s and compresses to tens of bits" in the revised paper. Soto, Florentina, Jen-Chun Hsiang, Rithwick Rajagopal, Kisha Piggott, George J. Harocopos, Steven M. Couch, Philip Custer, Josh L. Morgan, and Daniel Kerschensteiner. "Efficient coding by midget and parasol ganglion cells in the human retina." Neuron 107, no. 4 (2020): 656-666.

Zheng, Jieyu, and Markus Meister. "The Unbearable Slowness of Being." arXiv preprint arXiv:2408.10234 (2024).

We hope our responses answer your concerns.

Fair Comparisons With Baselines

Yes, I see that DeiT accuracy has increased over time due to improved training recipes. I’m worried that your dominance over other pruning baselines is for the same reason. If these baselines had the benefit of an improved recipe, would they be improved as well?

To put it in your own words, as line 415-417 states, "even with tricks that effectively retrained models, many pruning methods are not competitive... with modern baselines... which in essence are better trained ViTs". This seems like an issue, since we don’t know if your method is better since we don’t know how these other methods perform with modern recipes.

Choice of Little Network

I’m aware that there’s no restriction. That’s what makes me wonder why you choose to show only specific combinations of big and little network, rather than all of them.

In Table 2, EfficientNet-B2-288 uses EfficientViT as a little network, but most other EfficientNet variants do not. And EfficientNet-B2-288 is not used with EfficientNet-B1, but most other variants are. Unless there’s consistency or an obvious pattern in the choices made, the results look cherry-picked. Can you please explain the motivation behind the choice of combinations shown?

I increased my score 1 grade, as the authors have added a lot of empirical evaluation. As raised by another review, I do not think the novelty is strong, but I do think the empirical analysis is useful for understanding the relative accuracies of models and their behavior when combined in terms of efficiency/accuracy.

Expanded Contextualization of Novelty

As the ML community continues to produce increasingly large models with massive parameter counts, efficiently deploying them has become a significant challenge. Current inference optimization strategies, such as quantization or distillation, often require training new, smaller models or performing operations that result in performance drops. Additionally, these techniques fail to fully leverage the extensive ecosystem of models that is readily available (e.g., the EfficientNet family or user-submitted models on platforms like Torch or HuggingFace Hub).

In this context, we believe that our study on speeding up large models using smaller models is an important direction of work. Though the proposed protocol is simple to implement, our work holds significant practical value since

• our approach completely post-hoc, requiring no re-training.
• our approach is entirely model- and architecture-agnostic, allowing seamless integration with a variety of models.
• ours is the first work to systematically study and benchmark its utility in improving inference efficiency across a variety of tasks and model architectures.

By leveraging the growing diversity of pre-existing models across frameworks, Little-Big enables users to mix and match architectures (e.g., pairing models from different families such as EfficientNet and ViTs, BERTs and T5s). This flexibility ensures that one can adopt state-of-the-art models without the burden of additional training or specialized pipelines, and obtain significant compute savings and latency reductions without sacrificing accuracy. We argue that the framework's simplicity, adaptability, and compatibility with existing models make it a highly practical solution for real-world use cases.

Comparison to speculative decoding and cascaded systems

Speculative decoding (SpD) has been mainly developed for autoregressive sequence-to-sequence generative models where an output token is then used as a part of the input to the next inference step. With drafts from the smaller model, SpD decreases the large model latency by better utilizing the parallel computing capacity of modern hardware like GPUs, it actually increases the number of Flops. Little-big on the other hand operates in the predictive modeling domain and provides a net Flops reduction together with latency and throughput improvement.

While Little-Big indeed shares similarity with cascading strategies, these methods rely on task-specific architectures requiring training or fine-tuning at each stage for sequential re-ranking, Little-Big is a general protocol applicable across domains (e.g., vision, NLP, multimodal tasks) without task-specific modifications.

Additional Experiments

We thank you for this question. We want to first point out that in the main paper, we go beyond imagenet classification task and also consider the video-classification task (lines 485-497). Furthermore, we have now extended and studied Little-Big in four additional tasks. We refer the reviewer our summary response where we summarize our findings on extending our approach to various other tasks including text classification, semantic segmentation, multi-label classifiation and zero-shot classification tasks on a variety of benchmarks.

We hope our responses answer your questions and that you can recommend acceptance.

Dataset Diversity and Additional Experiments to Demonstrate Robustness

While the main paper focuses on the applicability of Little-Big to image and video classification (Kinetics-400), we have now extended our experiments, as suggested by the reviewer, to include additional datasets and settings to further demonstrate the robustness of our method.

We kindly request the reviewer to check our summary response on our expanded experimental evaluation (namely, zero-shot image classification, text classification, multi-label classification, and a pixel-level semantic segmentation). Details of the newly added experiments can be found in Appendix A.1 of the revised paper.

Threshold Selection and Generalization Concerns-

In Section 4.2 of the main paper, we address this question in detail. Below is a summary of our findings:

• Threshold Determination on Smaller Datasets: We evaluate the generalizability of the threshold T by determining it using a smaller validation set (e.g., ImageNet-V2) and transferring it directly to the larger ImageNet-1K dataset. We opted for this approach to ensure comparability with baselines evaluated on the full ImageNet-1K validation split. As shown in Appendix Figure 7, even when T is determined on ImageNet-V2, the Little-Big framework achieves significant MACs (>80%) reductions with no performance drop on ImageNet-1K.

• Cross-Dataset Threshold Transfer: We also examine the robustness of thresholds determined on the ImageNet-1K validation set by applying them directly to ImageNet-ReaL and ImageNet-V2. On both datasets, Little-Big achieves consistent MACs savings (>75%) with marginal accuracy losses of 0.04% and 0.07%, respectively.

• Threshold Determination for Non-ImageNet Tasks: In our newly added experiments (Section 7 and summary response), thresholds were determined using only 20% of the held-out validation data while performance and compute savings were then evaluated on the remaining 80%. Our results clearly demonstrate the generalizability of our approach across diverse modalities and tasks.

Related Work

We have expanded the related work section in the paper to include the additional methods suggested by the reviewer. Below, we summarize the key differences between these approaches and our proposed method:

• Early-Exit Models: Early-exit models reduce computation by allowing samples to exit at intermediate stages of the model pipeline. However, implementing these models is non-trivial and highly architecture-dependent. They also require training with specific objectives tailored to the early-exit mechanism. In contrast, Little-Big is architecture-agnostic, completely post hoc, and broadly applicable across multiple tasks, model families, and domains (e.g., images and text). Furthermore, we compare with Dynamic-Vit and A-ViT which are early-exit like models in Section 4.4 and demonstrate the benefits of Little-Big over these approaches.
• Confidence-Based Dynamic Routing: While similar to our approach in principle, confidence-based dynamic routing typically requires task-specific integration and training modifications. Little-Big differs itself by offering a simple thresholding mechanism that works across a wide range of pretrained models without requiring additional training.
• *Quantization: Quantization is indeed a relevant technique for model compression. While the focus of Little-Big is on inference-time compute savings without retraining, we acknowledge that quantization could further complement Little-Big. We have incorporated a short discussion of quantization in Section 2.3 and positioned it as an complementary method that can be integrated with Little-Big for additional gains. Furthermore,we highlight that the benefits of quantization are hardware dependent while Little-Big is a hardware independent approach.

Baselines Thank you for this question. In the A-ViT paper by Yin et al., the authors report the baseline DeiT performance as 78.9%, whereas the original DeiT paper reports a performance of 79.81% for the same model. Additionally, we confirmed the 79.81% performance through the publicly released DeiT-S checkpoint. Since it was not immediately clear which of these values to report, and to ensure fairness when comparing against other methods, we initially included both. However, we acknowledge that this choice could cause confusion for the reader. To address this, we have revised the paper to only include the 79.81% baseline from the original DeiT paper for clarity and consistency.
We also made similar changes in Table-3 with DeiT-B architecture to improve readability.

Thank you the author for their response. I was about to raise my scores since my concerns were addressed except I wanted to check the literature one more time from the survey I've shared earlier. I would have expected authors to do the same especially after my review. Looks like they either didn't do it, or somehow missed this paper:

https://dl.acm.org/doi/pdf/10.5555/2830840.2830854

From what I see it is pretty much this work, except it uses networks from 10 years ago. Given the existence of this work and lack of comparison in the current paper I lower my score.

We thank you for your review.

Additional Experiments We have now included additional experiments on all the tasks you suggested including - binary classification, multi-label classification, Zero-shot classification and semantic segmentation. Please refer to the our summary response and Sections A.1 in the updated paper.

In summary, across all the tasks, Little-Big consistently achieves significant MAC savings while comparable comparbale performance compared to the Big model.

Accuracy vs Latency and Online Classification

Thank you for raising this important point.

In the table below, we provide a comparison of change in the top-1 accuracy versus latency in a memory-constrained setup. In this scenario, we assume that both models cannot simultaneously fit in memory, requiring the Little model to be removed when the Big model is loaded. Even under these online classification constraints, Little-Big achieves high throughput and low latency while preserving the performance of the Big model.

It is worth noting that the latency and throughput numbers would further improve if the hardware had sufficient memory to fit both models simultaneously.

For more details, please refer to Section A.4 in the Appendix, titled "Benchmarking Throughput and Latency."

Novelty

As the ML community continues to produce increasingly large models with massive parameter counts, efficiently deploying them has become a significant challenge. Current inference optimization strategies, such as quantization or distillation, often require training new, smaller models or performing operations that result in performance drops. Additionally, these techniques fail to fully leverage the extensive ecosystem of models that is readily available (e.g., the EfficientNet family or user-submitted models on platforms like Torch or HuggingFace Hub).

In this context, we believe that our study on speeding up large models using smaller models is an important direction of work. Though the proposed protocol is simple to implement, our work holds significant practical value since

• our approach completely post-hoc, requiring no re-training.
• our approach is entirely model- and architecture-agnostic, allowing seamless integration with a variety of models.
• ours is the first work to systematically study and benchmark its utility in improving inference efficiency across a variety of tasks and model architectures.

By leveraging the growing diversity of pre-existing models across frameworks, Little-Big enables users to mix and match architectures (e.g., pairing models from different families such as EfficientNet and ViTs, BERTs and T5s). This flexibility ensures that one can adopt state-of-the-art models without the burden of additional training or specialized pipelines, and obtain significant compute savings and latency reductions without sacrificing accuracy. We argue that the framework's simplicity, adaptability, and compatibility with existing models make it a highly practical solution for real-world use cases.

We sincerely hope the new experiments and our responses answer your questions and you can champion our paper.

3. Semantic Segmentation

Next, we applied Little-Big to a more challenging pixel-level task -- semantic segmentation. For this experiment, we used the PyTorch—DeepLabV3 (MobileNet backbone, 11M parameters) as the Little model and the FCN (ResNet-101 backbone, 54M parameters) as the Big model. The evaluation was conducted using a subset of COCO with 20 categories overlapping with the Pascal VOC dataset. In this setup, we esimate the sample-level confidence as the lowest confidence among all superpixels ( superpixels are obtained using the SLIC algorithm), wherein the superpixel confidence was measured as the average of max. softmax probabilities from all pixels constituting the superpixel.

The result below further demonstrates the benefits of our approach.

4. NLP tasks

Finally, as requested by reviewers, we extended the Little-Big framework to NLP tasks, specifically focusing on text classification. We conducted experiments on the IMDB sentiment (binary) classification benchmark using DistilBERT (66M parameters) as the Little model and GPT-2 (124 parameters) as the Big model. The results, presented below, show that our approach achieves over 58% MACs reduction while maintaining the performance of the Big model. This demonstrates that Little-Big is not limited to vision tasks but is also effective in natural language processing domains.