LT-Soups: Bridging Head and Tail Classes via Subsampled Model Soups

We thank the reviewer for their positive feedback on the originality, robustness, and practical applicability of our method. Below, we address their concerns in detail.

Statistical Robustness: The paper lacks statistical significance measures such as error bars. Could you provide additional experiments or statistical tests to support the stability of your findings across multiple runs or data splits?

Please note that we followed the experimental protocol established in prior benchmarks [1,2]. While it is not feasible to run statistical tests across all datasets within the rebuttal timeline, we report results on TinyImageNet-LT using three different random seeds in the table below. As shown, the performance variance is negligible, supporting the robustness of our method. If the reviewers believe it would add value, we would be happy to include statistical tests on the remaining datasets in the final version of the paper.

[1] Liu et al. CVPR 2019. [2] Shi et al. ICML 2024.

Real-World Generalization: How does LT-Soups perform when tested on highly skewed datasets or under conditions with noisy labels or missing data? Would the approach still provide strong results?

Please note that two of the benchmark datasets—iNaturalist2018 and NIH-CXR-LT—are real-world datasets with naturally long-tailed distributions. They are characterized by markedly different values of imbalance ratio ($\rho$) and head-to-tail ratio ($\eta$): $\rho=500$, $\eta=0.11$ for iNaturalist2018 and $\rho = 6491$, $\eta = 5.66$ for NIH-CXR-LT. While incorporating additional distribution shifts, such as noisy labels or missing data, is a promising direction for future work, we follow prior work in the long-tailed learning literature [1,2] and focus solely on the imbalance aspect.

[1] Liu et al. CVPR 2019. [2] Shi et al. ICML 2024.

Alternative Subsampling Strategies: Have you considered experimenting with more advanced or adaptive subsampling techniques, such as those based on active learning or dynamic class balancing during training? Could they offer better performance than the current progressive subsampling strategy?

Thank you for the suggestion. We agree that more advanced subsampling strategies, such as selecting samples based on information content (e.g., based on entropy), could potentially improve performance and represent a promising direction for future work. However, in this study, we prioritize simplicity to keep our core message focused and interpretable. Notably, even our basic subsampling strategy already outperforms existing baselines, demonstrating the effectiveness of our overall approach.

We thank the reviewer for their thoughtful feedback and encouraging comments on our analysis, rigorous methodology, and well-designed ablation studies. We address their questions below:

W1. While LT-Soups is an effective framework, its core ideas are more of an intelligent combination of existing techniques.

We emphasize that our contribution goes beyond simply proposing a final model. Specifically:

1. A new lens for analyzing long-tailed learning: We introduce the head-to-tail ratio ($\eta$) as a complementary axis to the standard imbalance ratio ($\rho$), offering a more comprehensive view of class imbalance. We argue that effective long-tailed methods must perform well across the full ($\rho$, $\eta$) spectrum—a perspective that enhances understanding of real-world imbalance. Through this lens, we examine state-of-the-art methods such as a very strong competitor, LIFT [1], which advocates parameter-efficient fine-tuning (PEFT) for long-tailed tasks, and Model Soups [2], which uniformly average fully fine-tuned models on the full imbalanced dataset. This analysis reveals performance trade-offs that are often have overlooked in conventional evaluations.
2. LT-Soups (a principled adaptation of Model Soups): Building on these insights, we propose LT-Soups, which adapts the strengths of Model Soups to long-tailed settings. Model Soups boosts head-class performance but significantly degrades tail-class accuracy due to overexposure to head-heavy distributions. PEFT, on the other hand, limits parameter updates and struggles to adapt to head classes. LT-Soups addresses both issues by fully fine-tuning models on subsets with progressively increasing imbalance and combining them via recursive weight averaging. This balances head-class adaptation with tail-class generalization
3. Comprehensive evaluation: Across six benchmark datasets, LT-Soups outperforms PEFT in overall accuracy (by 1.3 to 2.3 percentage points) on five of them, while achieving comparable tail-class performance. It also consistently surpasses Model Soups in overall accuracy (by 1.3 to 2.3 points) and yields substantial gains in tail-class accuracy (by 2.6 to 10.9 points), with only a modest drop in head-class performance (up to 3.2 points).

[1] Shi et al. ICML 2025. [2] Wortsman et al. ICML 2022.

W2. A direct comparison of total GPU hours is missing, making it difficult to assess the practical cost-benefit trade-off.

The full computational analysis, including total wall-clock time for the parallel version of LT-Soups, is already provided in Appendix B. In response to the reviewers’ request, we also report the total GPU hours, assuming access to only a single GPU with modest memory.

To recap, LT-Soups consists of two stages: Stage 1 trains a series of specialist models on subsampled datasets with varying imbalance ratios. Stage 2 involves retraining only a linear classifier on the full dataset, which is computationally lightweight. Both stages use the same number of epochs.

Tables 1 and 2 report the Stage 1 fine-tuning time per subsample (each $\rho$ corresponds to one subsample) and the total GPU hours when LT-Soups is trained on a single GPU for the ImageNet-LT and NIH-CXR-LT datasets. As we can see, the computational cost of LT-Soups depends primarily on the degree of imbalance in the dataset. For example, in ImageNet-LT (which comes with \rho=256), following our subsampling strategy, the total absolute GPU time is ~8 hours. LT-Soups is costly in this case because with our largest \rho=64, it includes 80% of the entire training data. In contrast, in NIH-CXR-LT ($\rho = 6401$), the largest subset ($\rho = 256$) includes only 24% of the data, resulting in substantial savings, up to 18x lower GPU hours compared to Model Soups of the same size.

Table 1: Computational cost per ImageNet-LT subset for LT-Soups with $M=2$.

Table 2: Computational cost per CXR-LT subset for LT-Soups with $M=2$.

We conclude by referencing Table 9 in the Appendix, which compares the total computational cost of LT-Soups and Model Soups (both with parallelization) against other baselines on ImageNet-LT and NIH-CXR-LT. Notably, although LT-Soups is a full-rank method, it converges within 10 epochs on CXR-LT, while LoRA—despite its parameter efficiency—requires 50 epochs, resulting in longer wall-clock time and higher total GPU hours. Thus, in response to the reviewer’s concern, the claim that “the total computational budget is significantly higher than single-run methods” does not always hold; LT-Soups often reduces computational cost through subsampling, with the budget depending on the choice of $N$ (Tab. 8 Appendix).

W3. The paper lacks a deep, intuitive explanation for why this curriculum-like fusion (from balanced to imbalanced) is the optimal approach.

We believe our recursive weight averaging (WA) is a principled approach, as it enables an effective trade-off between stronger adaptation—potentially biased toward head classes—and more balanced generalization. It can be interpreted as an exponential moving average (EMA) over fine-tuned models sorted by increasing imbalance severity, with a tunable parameter that adjusts the influence of more balanced (but smaller) versus less balanced (but larger) subsets. In contrast, uniform WA applies a simple arithmetic mean, giving equal weight to all models regardless of their imbalance level.

In all of our experiments in the paper, we use only two values for $\lambda$: 0.3 and 0.7, corresponding to high and low adaptation needs, respectively. Intuitively, when the target dataset is close to the pre-training weights, the value of the $\lambda$ becomes less important as even small datasets are enough for adaptations. However, when the shift becomes larger, subsets with more data (albeit biased towards head classes) become crucial.

The table below confirms our hypotheses. In particular, we compare recursive WA and uniform WA across two datasets with different similarities compared to CLIP-pretrained weights (according to the zero-shot performance). On ImageNet-LT, which is already well-aligned with CLIP-pretrained features, there is little to no difference between the two averaging schemes. However, for datasets that require significant adaptation [1], such as iNaturalist2018, recursive WA yields clear benefits by leveraging information from more data-rich subsets.

Table 1: Performance comparison under uniform and recursive WA across three datasets.

W4. Lack of discussion on recent literature.

Thank you for the suggestions. Given that our focus was on using pre-trained CLIP models for LT recognition, we have restricted the comparison to the recent literature that uses CLIP for LT recognition. In response to the reviewer’s request, we have expanded the related work section and now provide a brief comparison of LT-Soups with the methods mentioned.

SADE [1], Mdcs [2], and DirMixE [5] are Mixture-of-Experts (MoE) methods that aggregate diverse experts trained with different logit adjustment (LA), targeting distributions like uniform, long-tail, and inverse long-tail. Unlike these methods, which require deploying all experts at inference, LT-Soups uses weight averaging to collapse models into a single, inference-efficient solution. Moreover, as noted in [6], fully fine-tuning foundation models like CLIP with LA loss alone is insufficient, as it can lead to inconsistent class-conditional distributions, particularly for tail classes.

Reflective Learning [3] promotes consistency across training iterations by minimizing KL divergence between predictions and soft labels induced from feature similarity. In this light, the EMA mechanism in LT-Soups can be viewed as a lightweight form of Reflective Learning or self-distillation [7].

[1] Zhang et al., NeurIPS 2022. [2] Zhao et al., ICCV 2023. [3] Zhao et al., ECCV 2024. [5] Yang et al., ICML 2024. [6] Shi et al., ICML 2024. [7] Allen-Zhu et al., ICLR 2023.

I thank the authors for their detailed and timely response. After reviewing their feedback, I believe that most of my concerns have been adequately addressed. I have therefore decided to raise my original rating, leaning toward acceptance.

We thank the reviewer for their insightful feedback and for recognizing the simplicity and effectiveness of our work. Our responses are below:

Q1. Performing PEFT and Model soups, followed by performing the same final layer only tuning with logit adjustment that is performed in LT-Soups -- would this help in closing gap in head accuracy for PEFT/Model soups ?

We found that additional final-layer tuning with logit adjustment on PEFT and Model Soups has little to no effect. The table below summarizes the results on TinyImageNet-LT. We hypothesize that, unlike these baselines, LT-Soups does not fully exploit the entire training set, due to the downweighting effect introduced by weight averaging. Consequently, fine-tuning the final layer helps LT-Soups recover head-class sharpness and improves overall performance.

Table 1: Comparison of baselines with and without Classifier Re-training

Q2. Take a subsampled dataset that has the least imbalance, perform vanilla model soups on it or PEFT, followed by final layer only tuning -- this ablation would tell us that does one needs multiple models at different levels of imbalance in the dataset, or does the least imbalanced dataset is enough

Table 3 in the main paper has already addressed this question in the context of Model Soups. Per your request, we extend the same experiment to the PEFT baseline, with full results shown in the table below.

Specifically, we compare LT-Soups with Model Soups and PEFT baselines that follow the same two-stage framework as LT-Soups. The only difference lies in the first stage: PEFT trains a single model on a subset with a fixed imbalance ratio, while Model Soups averages 16 models, all trained on subsets sharing the same imbalance ratio.

Results show that performance varies significantly depending on the imbalance ratio used. For instance, Model Soups with $\rho = 8$ yields the highest tail accuracy (75.0), while using the full dataset ($\rho = 100$) results in the highest head accuracy (85.9). A similar trade-off is observed with PEFT. However, under any single imbalance setting, both baselines fall short of LT-Soups, which achieves performance of 78.6 (overall), 85.0 (head), and 75.2 (tail).

Rather than optimizing for a single point on the imbalance spectrum, LT-Soups averages across subsets with varying imbalance levels. This enables it to integrate the strengths of both low- and high-$\rho$ models, resulting in a more balanced head–tail trade-off overall.

We thank the reviewer for their feedback and address their concerns in detail below:

W2. It seems that the proposed method cannot solve the targeted problem. From the experiments, the proposed methods can improve the performance of model soup on tail classes, but the performance on head classes still degrades which is the problem that the authors stated to solve.

It is well established in the class imbalance literature [1,2,3,4] that, given an imbalanced dataset, the goal is to maximize balanced accuracy (see L122–124) on a uniformly distributed test set. This, in turn, requires achieving a well-balanced performance across both head and tail classes.

While Model Soups favours head-class performance by fully fine-tuning on the original imbalanced distribution, this improvement comes at the cost of reduced accuracy on tail classes. On the other hand, PEFT limits weight updates to a small subset of parameters, which negatively impacts head-class performance. In contrast, LT-Soups fine-tunes and averages models trained on distributions with progressively increasing imbalance ratios. This strategy not only allows adaptation to head classes but also preserves the balanced representations needed for tail-class generalization. Across all six benchmarked datasets, LT-Soups consistently outperforms Model Soups in overall accuracy (by 1.3 to 2.3 percentage points) and shows substantial improvements in tail-class accuracy (by 2.6 to 10.9 points), with only a modest decrease in head-class performance (up to 3.2 points).

[1] Cao et al. Neurips 2019. [2] Ren et al. Neurips 2020. [3] Menon et al. ICLR 2020. [4] Wang et al. ICLR 2020.

W3. The experimental setting is not comprehensive enough. For example, one of the setting of imbalance ratio or head-tail ratio in Table 1 and 2 should be kept when the other one is changing.

Please note that we followed the experimental protocol established in prior benchmarks [1,2]. Particularly, we conduct a comprehensive analysis across six datasets: four synthetic long-tailed distributions, CIFAR100-LT, ImageNet-LT, TinyImageNet-LT, and Places-LT, and two real-world long-tailed distributions, NIH-CXR-LT and iNaturalist2018. Each dataset features a distinct combination of imbalance ratio and head-to-tail ratio, providing a diverse representation of imbalance scenarios.

Furthermore, in Section 3.2 of the paper, we systematically vary the imbalance ratio ($\rho$) across three levels and the head-to-tail ratio ($\eta$) across eleven values by synthetically modifying the CIFAR-100 distribution. Each configuration is evaluated across four baselines, including LT-Soups, resulting in a total of 396 training runs. While we expect the trends observed on CIFAR-100 to generalize to other datasets such as TinyImageNet-LT and those in Tables 1 and 2, we are happy to include additional results in the Appendix if the reviewer believes it would strengthen our findings.

[1] Liu et al. CVPR 2019. [2] Shi et al. ICML 2024.

W4. Can you show how the performance changes when the total number of models changes?

Increasing the total number of models consistently improves overall performance, as it has already been presented in Figure 5b and L296 in the main paper and Table 7 in the Appendix. For the reviewer’s convenience, we summarize these results in the two tables below.

To recap, the total number of models used in LT-Soups is governed by two key hyperparameters: $N$, the number of subsamples with varying imbalance ratios, and $M$, the number of models trained per subsample.

Table 1 explores the effect of increasing the total number of models by varying $N$, from the most balanced case ($N=1$) to the full dataset ($N=8$) while keeping the number of models per subsample fixed at $M=2$. Table 2 examines the impact of increasing $M$ while holding $N=8$ fixed. Overall, we observe consistent improvements in performance as the total number of models increases. To keep the experiments manageable, we fix $M = 2$ in the main benchmarks.

Table 1: Effect of increasing the total models on performance by varying the number of subsamples ($N$)

Table 2: Effect of increasing the total models on performance by varying the number of models trained per subsample ($M$)

W1.1. The results illustrated in Figure 2 is not convincing enough because they are averaged on head-tail ratio which will not be able to show the detailed results when the head-tail ratio is high or low.

We would like to clarify that the analysis requested by the reviewer corresponds to Figure 1 of the main paper. This figure reports the performance of four methods, including LT-Soups, on synthetic CIFAR100 across three levels of imbalance ratio ($\rho$) and eleven head-to-tail ratio ($\eta$) values. Figure 2 is derived from the same data by marginalizing over both $\rho$ and $\eta$ to emphasize a central insight of our work: an effective LT method must perform well across the full spectrum of imbalance conditions.

W1.2. Throughout the paper, how the Logit Adjustment is performed is not mathematically formulated. Although it has been proposed in previous works, rephasing it in the current paper helps others understand it better.

We appreciate your suggestion and will revise the description of the Logit Adjustment loss function in Section 3.1 for greater clarity.

W1.3. The caption of Figure 5 is neglected as well.

To improve clarity, we split the original Figure 5 into two separate figures, each with its own caption.

W1.4. The momentum coefficient $\mu$ in line 210 is not defined either. If it is the same as $\lambda$ above, then the last weight of the averaged model is almost the same as the first model, so that the proposed method is meaningless.

We use the following formulation for EMA:

$\theta_{ema} = (1-\mu) \cdot \theta_{ema} + \mu \cdot \theta_0,$

where $\theta_0$ is the initialization weights and $\theta_{ema}$ denotes the fine-tuning parameters. We fixed $\mu=0.01$ throughout all the experiments. Therefore, the last weight of the averaged model is clearly not the same as the first model. We will clarify this in the revised manuscript.

Thanks to the authors' responses. Most of my problems have been solved. Sorry for that I overlooked some parts of the paper and had some misunderstanding. I have raised my rating to recommend acceptance.