Author Response to Reviewer VDfk

We thank the reviewer for the thoughtful review and the constructive feedback.

[W1] Direct Comparison between Asymmetric Duos (AD) and Deep Ensembles (DE).
We thank the reviewer for highlighting this important comparison. Here we directly compare ADs and ResNet-50 DEs on ImageNet while controlling for total FLOPS. In short, while Deep Ensembles show diminishing returns as the number of members ($m$) increases, Duos continue to improve in accuracy and uncertainty quantification with higher FLOPS. This makes Duos more effective under high compute budgets.

We provide the results and experimental details below:

This comparison highlights the superior scalability of Asymmetric Duos under FLOPS budgets. AD$[m,m+1)$ denotes Asymmetric Duos with total FLOPS within $m$ to $m+1$ times of that of a Base ResNet50 Model. Note that all metrics values were scaled by $10^{2}$ for better presentation, with the exception of Brier score which was scaled by $10^{4}$. DEs of sizes $m=2$ to $8$ are formed by 32 random subsets of 10 ResNet-50 checkpoints from timm; ADs compose a ResNet50/ResNet101/ResNet152/ConvNeXt-Base/ViT-B base model with a sidekick model with FLOPS Balances $\geq0.1$.

We also repeated this analysis controlling for parameter count instead of FLOPS, and observed similar trends. These results and their visualizations will be included in the Appendix in our revision. These results and their visualizations will be included in the Appendix in our revision. We hope our findings on Asymmetric Duos would spark broader interest in model combination methods, including homogeneous DEs, and this direct comparison would contribute to a better understanding of when and how to apply each approach effectively.

[W2] Incremental Novelty.
We appreciate the reviewer's perspective, but note the empirical novelty of our heterogenous Duos, that can effectively combine two fine-tuned networks of vastly different sizes without joint fine-tuning. Asymmetric Duos are uniquely poised to be highly compatible with today's large pretrained models and fine-tuning workflow, and are able to "interpolate" the accuracy-FLOPs trade off by adding fractional additional computation w.r.t. to the base model, introducing finer control over the scaling laws at minimal computational cost.

[Q1] Scales of Un-normalized Logit Vectors.
This point on a potential scale mismatch of raw logits is easy to overlook, we thank the reviewer for the careful observation.

You're right that naively averaging logits from heterogeneous models can introduce bias due to norm mismatches; an issue we also observed in the Unweighted (ablation) Duos. Our AD strategy specifically addresses this through temperature-based logit scaling. Since scaling logits by a temperature also scales their outputs' norm linearly, the learned temperatures implicitly adjust the scale between model outputs. For instance, on the ImageNet validation set, EfficientNet-V2-L (145M parameters) and EfficientNet-B0 (5M parameters) have average logit vector l2 norms of 24.50 and 33.07, respectively. Our learned temperatures (1.64 for V2-L and 0.86 for B0) adjust these to 40.21 and 28.31, effectively up-weighting the base model’s contribution.
We observe similar results for all Duos on all datasets, and will include this discussion in the revision to alleviate concerns around this point.

We thank the reviewer again for the thoughtful feedback. We hope our responses adequately address the concerns raised and would be happy to discuss further during the reviewer–author discussion period.

We thank the reviewer for their quick and thoughtful response. We're encouraged that our rebuttal helped reinforce the reviewer's positive assessment of our work.

We also appreciate the clarification regarding the Deep Ensemble comparison. We agree that this base-model-controlled setup is a meaningful and practical complement to our FLOPs-controlled comparison. Both offer useful perspectives, and we now include results under the reviewer’s requested setting using ResNet50 (RN50) for all Ensembles members and Duos $f_{\text{large}}$.

Here, Deep Ensembles follow the “pool-then-calibrate” strategy from [Rahaman2021]. Since our temperature-based Duo aggregation inherently performs calibration, this enables a fairer comparison on NLL, Brier, and ECE. All ensembles are drawn from random subsets of 10 independently trained ResNet50 (RN50) checkpoints (from both timm and torchvision). As requested, Duos in this setting are $f_{\text{large}}$-controlled: each Duo shares a fixed RN50 as $f_{\text{large}}$, and varies its sidekick among MNASNet0.75 (MN.75), ShuffleNet V2 2.0× (SN2.0x), and EfficientNet-B2 (ENB2), with FLOPs Balances (FB) of 0.05, 0.14, and 0.27, respectively.

Despite their lower compute, these asymmetric RN50 Duos quickly approach performance of larger, homogeneous Deep Ensembles (e.g., $m=5$ RN50s) across most metrics. We attribute this to architectural diversity from heterogeneity [Gontijo-Lopes2022] and Duos' simple but effective temperature-based aggregation. Together, these allow RN50-based Duos to realize a large portion of the performance gains of full $m=5$ Deep Ensembles at a fraction of the cost.

We hope this base-model-controlled comparison directly addresses the reviewer’s concern. We have updated the manuscript to include both the FLOPs-controlled and $f_{\text{large}}$-controlled Duo comparisons in a new Appendix section titled “Evaluating Asymmetric Duos against Deep Ensembles”. There, we note that ensembling remains a strong approach in uncertainty-critical scenarios, and Duos offer an alternative that retains much of the accuracy and UQ benefits. Relevant pointers have been added to both the main Experiments and Discussion sections.

Note that we are including the same response for Reviewer ZFrX, who raised a similar concern. We appreciate this alignment, as it clearly signals the value of including this comparison in our paper.We believe this addition makes our analysis more thorough and helps better position Duos among other methods for practitioners, as the reviewer suggested. We thank the reviewer for this thoughtful suggestion on direct comparisons between DEs and $f_{\text{large}}$-controlled ADs.

References

[Rahaman2021] Rahul Rahaman and Alexandre H Thiery. Uncertainty quantification and deep ensembles. Proceedings of the 35th International Conference on Neural Information Processing Systems, 2021.

[Gontijo-Lopes2022] Raphael Gontijo-Lopes, Yann Dauphin, and Ekin Dogus Cubuk. No one representation to rule them all: Overlapping features of training methods. International Conference on Learning Representations, 2022.

I would like to thank the authors for the results, which I believe make the experimental evaluation more complete now. The findings look solid and allow to place the proposed assymetric duos nicely in the landscape of other models. I appreciate that you were able to deliver these results within the discussion period, good work!

My questions and concerns are addressed, I will continue to monitor the other reviewer discussions to make sure I did not miss anything, but will then update my final score.

Author Response to Reviewer ZFrX

We thank the reviewer for the thoughtful review and the insightful suggestions.

[W1&Q2] Direct Comparison between Asymmetric Duos (AD) and Deep Ensembles (DE).
We thank the reviewer for highlighting this important comparison. Here we directly compare ADs and ResNet-50 DEs on ImageNet while controlling for total FLOPS. In short, while Deep Ensembles show diminishing returns as the number of members ($m$) increases, Duos continue to improve in accuracy and uncertainty quantification with higher FLOPS. This makes Duos more effective under high compute budgets. We provide the results and experimental details below:

This comparison highlights the superior scalability of Asymmetric Duos under FLOPS budgets. AD$[m,m+1)$ denotes Asymmetric Duos with total FLOPS within $m$ to $m+1$ times of that of a Base ResNet50 Model. Note that all metrics values were scaled by $10^{2}$ for better presentation, with the exception of Brier score which was scaled by $10^{4}$. DEs of sizes $m=2$ to $8$ are formed by 32 random subsets of 10 ResNet-50 checkpoints from timm; ADs compose a ResNet50/ResNet101/ResNet152/ConvNeXt-Base/ViT-B base model with a sidekick model with FLOPS Balances $\geq0.1$.

We also repeated this analysis controlling for parameter count instead of FLOPS and observed similar trends. These results and their visualizations will be included in the Appendix in our revision. We hope these findings will spark broader interest in methods for both homogenous combination (ensembles) and heterogeneous combination (asymmetric Duos) and contribute to a better understanding of when and how to apply each approach effectively.

[W2] Scalable Bayesian Methods.
We thank the reviewer for making thoughtful and specific suggestions on scalable bayesian methods.

Improved Variational Online Newton (IVON)

We benchmark Asymmetric Duos against a ResNet50 fine-tuned with IVON optimizer on Caltech256.

IVON works well out of the box, outperforming standard AdamW with minimal training overhead. Its sampling-based inference further improved over the deterministic version, though like Deep Ensembles, it introduces an $m\times$ test-time cost with diminishing returns. In contrast, Asymmetric Duos built on ResNet-50 achieve better performance across all metrics with only a fractional increase in compute. Furthermore, similar to the compatibility shown between soup and Asymmetric Duos in Section 5.5, IVON, as a great plug-and-use uncertainty-aware optimizer, would fit nicely with our Asymmetric Duos framework.

Laplace Approximation

We also compare Laplace Approximation (LA) with Asymmetric Duos. Due to LA's higher computational and memory usage when applied to architectures with high last-layer dimension, we perform this analysis on Caltech256 using a Swin-T model.

We observe that, similar to IVON, Laplace Approximation (last-layer-only & kron & probit as recommended) reliably improves over the base model. However, it falls short in absolute performance gains compared to Asymmetric Duos across most metrics. Notably, LA can also be incorporated into any Duo, further demonstrating the generality and practicality of the Asymmetric Duos framework.

[W3] More Asymmetric Duos Aggregation.
We thank the reviewer for recognizing that the simple fixed temperature technique is effective. We argue this simplicity is a strength: if we select temperature weights to maximize validation accuracy, Duos necessarily outperform the large model on validation data (the large model is simply a Duo with weights [1,0]). Given the low VC dimension of the space of possible temperature weightings (≤3), standard learning theory provides O($\sqrt{1/|\mathrm{val_set}|}$) generalization bounds on accuracy improvement with small constants. We will include a derivation of this bound in revision (and we can provide details to the reviewers if needed, though it follows from standard learning theory). Second, the efficacy of such a simple approach is itself a notable empirical contribution. While established work suggests that ensembling models of different performance levels typically yields degradations [Opitz1999; Wood2023], our finding that asymmetric models can be effectively combined with simple temperature weighting should surprise the community. Again, we will include this discussion in the revision. While data-dependent mechanisms could yield further gains, they would sacrifice these theoretical guarantees and require additional training overhead. Nevertheless, we believe pursuing such "Dynamic Duos" is a fruitful direction for future work.

[Q1] Temperature behaviors in practice.
Learned temperatures tend to emphasize the base model in low-FLOPS-balance regimes and gradually increase the weight on the sidekick model as the FLOPs balance improves. Figure 10 in Appendix D illustrates this behavior for Asymmetric Duos, showing the ratio between the learned temperatures of the base and sidekick models as a function of FLOPs balance. This reveals a gradual shift in weighting toward the sidekick as its relative capacity increases.

[Q3] Bayesian Model Averaging and Asymmetric Duos.
The connection between Duos and Bayesian model averaging is interesting. If one were to combine posterior samples from a large model and a small model, this could be interpreted as sampling from a hierarchical Bayesian model where the prior is a mixture over architecture types, effectively placing prior probability mass on both small and large model architectures. We agree that a Bayesian version/interpretation of Duos is an interesting avenue for future work that could make use of additional train-time or test-time computation beyond our efficient duos.

We thank the reviewer again for the thoughtful feedback. We hope our responses adequately address the concerns raised and would be happy to discuss further during the reviewer–author discussion period.

References

[Opitz1999] Opitz, R. M., & Maclin, D. W. (1999). Popular ensemble methods: An empirical study. Journal of Artificial Intelligence Research, 11, 169–198.

[Wood2023] Wood, D., Mu, T., Webb, A., Reeve, H. W. J., Luján, M., & Brown, G. (2023). A Unified Theory of Diversity in Ensemble Learning. Journal of Machine Learning Research, 24, 1–49.

We thank the reviewer for their quick and thoughtful response. We're encouraged that our rebuttal helped reinforce the reviewer's positive assessment of our work.

We also appreciate the clarification regarding the Deep Ensemble comparison. We agree that this base-model-controlled setup is a meaningful and practical complement to our FLOPs-controlled comparison. Both offer useful perspectives, and we now include results under the reviewer’s requested setting using ResNet50 (RN50) for all Ensembles members and Duos $f_{\text{large}}$.

Here, Deep Ensembles follow the “pool-then-calibrate” strategy from [Rahaman2021]. Since our temperature-based Duo aggregation inherently performs calibration, this enables a fairer comparison on NLL, Brier, and ECE. All ensembles are drawn from random subsets of 10 independently trained ResNet50 (RN50) checkpoints (from both timm and torchvision). As requested, Duos in this setting are $f_{\text{large}}$-controlled: each Duo shares a fixed RN50 as $f_{\text{large}}$, and varies its sidekick among MNASNet0.75 (MN.75), ShuffleNet V2 2.0× (SN2.0x), and EfficientNet-B2 (ENB2), with FLOPs Balances (FB) of 0.05, 0.14, and 0.27, respectively.

Despite their lower compute, these asymmetric RN50 Duos quickly approach performance of larger, homogeneous Deep Ensembles (e.g., $m=5$ RN50s) across most metrics. We attribute this to architectural diversity from heterogeneity [Gontijo-Lopes2022] and Duos' simple but effective temperature-based aggregation. Together, these allow RN50-based Duos to realize a large portion of the performance gains of full $m=5$ Deep Ensembles at a fraction of the cost.

We hope this base-model-controlled comparison directly addresses the reviewer’s concern. We have updated the manuscript to include both the FLOPs-controlled and $f_{\text{large}}$-controlled Duo comparisons in a new Appendix section titled “Evaluating Asymmetric Duos against Deep Ensembles”. There, we note that ensembling remains a strong approach in uncertainty-critical scenarios, and Duos offer an alternative that retains much of the accuracy and UQ benefits. Relevant pointers have been added to both the main Experiments and Discussion sections.

We are including the same response for Reviewer VDfk, who raised a similar concern. We appreciate this alignment, as it clearly signals the value of including this comparison in our paper. We believe this addition makes our analysis more thorough, and thank the reviewer for this thoughtful suggestion on direct comparisons between DEs and $f_{\text{large}}$-controlled ADs.

References

[Rahaman2021] Rahul Rahaman and Alexandre H Thiery. Uncertainty quantification and deep ensembles. Proceedings of the 35th International Conference on Neural Information Processing Systems, 2021.

[Gontijo-Lopes2022] Raphael Gontijo-Lopes, Yann Dauphin, and Ekin Dogus Cubuk. No one representation to rule them all: Overlapping features of training methods. International Conference on Learning Representations, 2022.

Author Response to Reviewer x6b8

We thank the reviewer for the review.

[Q1 & Q2] Aggregation Strategy Confusion

There may be some confusion about our core contribution. While our method uses an aggregation approach that is indeed standard validation-based weighting, our core contribution is applying this standard technique to models of vastly different accuracies to achieve the performance improvements of ensembles at fractional computational costs. This finding is surprising given that practical wisdom and prior work [Opitz1999; Wood2023] suggest asymmetric model combinations should not work effectively. We have added this clarification to Section 6 (Limitations and Future Directions) to better highlight our contribution, and we hope this resolves the confusion regarding the aggregation strategy.

This work serves as an initial step toward understanding the aggregation strategies of deep networks of different sizes and capacities. While we focused on a small set of interpretable aggregation strategies for both point prediction and uncertainty quantification, there remains a large space of unexplored methods. Notably, our finding that simple validation-based weighting can yield effective ensembles from models with vastly different accuracies is itself a surprising and underexplored result.

[Q3] Asymmetric Duos compared to SOTA methods

We are not sure what SOTA methods the reviewer is referring to. We built Asymmetric Duos directly off of popular and recent models that have been empirically demonstrated to be consistently best-performing through extensive evaluations by [Goldblum2023] on a suite of computer vision tasks. We additionally compare and combine Asymmetric Duos to recent SOTA methods like model soup [Wortsman2022] and show compatibility in Section 5.5. We would be happy to receive more specific feedback on which methods the reviewer is interested in seeing us benchmark against.

References

[Goldblum2023] Micah Goldblum, Hossein Souri, Renkun Ni, Manli Shu, Viraj Prabhu, Gowthami Somepalli, Prithvijit Chattopadhyay, Mark Ibrahim, Adrien Bardes, Judy Hoffman, et al. Battle of the backbones: A large-scale comparison of pretrained models across computer vision tasks. Advances in Neural Information Processing Systems, 36:29343–29371, 2023.

[Opitz1999] Opitz, R. M., & Maclin, D. W. (1999). Popular ensemble methods: An empirical study. Journal of Artificial Intelligence Research, 11, 169–198.

[Wortsman2022] Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: Averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Learning, pages 23965–23998. PMLR, 2022.

[Wood2023] Wood, D., Mu, T., Webb, A., Reeve, H. W. J., Luján, M., & Brown, G. (2023). A Unified Theory of Diversity in Ensemble Learning. Journal of Machine Learning Research, 24, 1–49.

Author Response to Reviewer 72FX

We thank the reviewer for the thoughtful review and insightful questions.

[W1] Regarding theoretical foundations
We appreciate this feedback. While we focused on empirical validation given the practical nature of our contribution, we could easily establish several straightforward theoretical results.

Accuracy improvement guarantee
If we select temperature weights to maximize validation accuracy, Duos necessarily outperform the large model on validation data (the large model is simply a Duo with weights $[1,0]$). Given the low VC dimension of the space of possible temperature weightings ($\leq 3$), standard learning theory provides $O\left(\sqrt{1/|\mathrm{val\_set}|}\right)$ generalization bounds on the test accuracy improvement.

Ensemble equivalence
Recent work [Dern2025, Abe2022] show large models behave like ensembles of smaller models. A direct application of Theorem 3.3 from Dern2025 proves that a Duo combining M×D and D-parameter random feature models is equivalent to an ensemble of $(M+1)$ D-parameter models—formalizing our intuition about why asymmetric combinations work.

Both proofs are approximately 10 lines and follow from standard techniques. We omitted them to avoid cluttering the paper, but we're happy to include whichever result you feel would most benefit readers. Would either theorem meaningfully strengthen the contribution in your view?

[W2] Practical Selection of Duos
We thank the reviewer for this thoughtful point regarding the practicality of selecting the sidekick model in an Asymmetric Duo, a question that interests us as well.

Similar to choosing a backbone model for fine-tuning, it is indeed difficult to know beforehand which Duo will perform best on a specific downstream task. That said, some heuristic indicators have been proven useful. In particular, we have found that FLOPs balance and pre-training performance on a standard dataset like ImageNet are informative when choosing the sidekick model. FLOPs balance is free and reliable, and is aligned with the general "larger models perform better" intuition in modern ML. Lastly, as the review suggested, most Duos lead to performance improvements, so even though it's hard to predict the optimal choice of sidekick, it is easy to find good contenders.

[Q1] Training Dataset
Yes, both the base model and the sidekick model in our experiments have been trained on the same downstream training set, though this is not mandatory. We can still make Duos even if a sidekick model has only seen partial or none of the entire training set, and this would be an interesting investigation for future work.

[Q2] Adding more than one sidekick model
We thank the reviewer for pointing out this natural follow-up of our Duo framework.

Combining a large base model with more than one sidekick can be considered as another approach to scale up the Asymmetric framework, and its efficiency/performance tradeoff is one that we find interesting as well. We conducted additional experiments using a Trio setup, aggregating a base model with two sidekicks using the same temperature weighting strategy described in Section 3.1, while keeping the total additional FLOPs within 20%. While Trios sometimes show slight improvements over Duos in AUROC and AURC, we also observe some regressions in Accuracy, Brier score, and NLL.

Overall, we find that Asymmetric Duos offer a more consistent and efficient tradeoff at low FLOPs budgets, making them the more practical choice in most settings.

References

[Abe2022] Abe, Buchanan, Pleiss, Zemel, Cunningham. Deep Ensembles Work, But Are They Necessary? NeurIPS 2022.

[Dern2025] Niclas Dern, John P. Cunningham, and Geoff Pleiss. Theoretical limitations of ensembles in the age of overparameterization ICML 2025.