RSMerge: Bridging Head and Tail Classes via Subsampled Model Merging

We thank the reviewer for the thoughtful feedback and positive comments on our novel findings, rigorous methodology, and thorough empirical analyses. We address the questions as follows:

---

Q1. I would encourage the authors to include a README explaining how to install/run the code.

We want to highlight that we have included a run.sh script for reproducibility and will add a detailed README with step-by-step instructions upon the official code release.

---

Q2. Computational analysis.

Thanks for the suggestion. The table below compares the computational costs of Full Fine-Tuning (Full-FT), LIFT (which employs a LoRA adapter with rank 64 applied to all MLP layers), and RSMerge on the ImageNet-LT and CXR-LT datasets. All models were trained to convergence using a batch size of 128 and mixed-precision training. For RSMerge, we break down the computational cost into two stages. Stage 1 involves training models independently and in parallel on subsets with different imbalance ratios. Since the subset with the highest imbalance ratio contains the most training samples, it dominates the overall wall-clock time. Stage 2 retrains only the linear classifier on the full dataset—a highly efficient step, as it updates just a single linear layer. In our experiments, we used the same number of epochs for both stages of RSMerge.

The computational overhead of RSMerge compared to existing methods depends heavily on dataset characteristics—particularly the original imbalance ratio. For example, in ImageNet-LT, which has an imbalance ratio of 256, the largest subset used in Stage 1 accounts for 89% of the full training data, resulting in relatively higher wall-clock time. In contrast, on CXR-LT, with a much more extreme imbalance ratio of 6401, the largest Stage 1 subset represents only 24% of the dataset, leading to a 4.4× reduction in training time compared to Full-FT (see Table 7 in the appendix for details). Additionally, while full-rank methods like Full-FT and RSMerge typically converge within 10 epochs on CXR-LT, LIFT required 50 epochs—substantially increasing its wall-clock time despite its parameter-efficient design.

---

Q3. A compute statement is missing, e.g., concerning the hardware used to run the experiments and/or total cost in cloud compute credits.

All experiments were conducted on a cluster equipped with NVIDIA RTX 3090 GPUs (24GB VRAM), using Python 3.9.15, PyTorch 2.4.0, and CUDA 11.8.

---

Q4. Which hyperparameters are tuned on the validation set? In the paper, only temperature (for the calibration study) and early-stopping duration are mentioned. More specifically, how were $\lambda$, $M$, and $N$ chosen?

Across all datasets, we fix $M=2$. As discussed in Section 5.2, we validate two $\lambda$ values (0.3 and 0.7): datasets closely aligned with CLIP's pre-trained distribution benefit from a higher $\lambda$, while others perform better with a lower value. For $N$, we use imbalance factors that are powers of 2, ensuring the maximum factor is less than half of the full dataset's imbalance (see Table 7). We use the same number of epochs as in LIFT—except for CXR-LT, where hyperparameter search spans all baselines. Finally, we use an equal number of epochs for both RSMerge stages.

---

Q5. Other Comments Or Suggestions.

Per the reviewers’ request, we have implemented the suggested changes, including ensuring consistent usage of periods after \paragraph, correcting citation formatting (L112/114), fixing the backward quotation mark (L163), replacing << with \ll (L189), removing the extra space after "(Figure 5)" (L199), correcting the typo in "Classification" (L291), and fixing the malformed sentence in L322 ("Table 1 An").

We thank the reviewer for their insightful feedback and for recognizing the value of our work on imbalanced training. Our responses to their questions, which we found fruitful and led to several new experiments and analyses, are below:

---

Q1. It would be valuable to see if the weight averaging of the models provides performance improvements regardless of the subsets' design (as would happen with traditional ensemble methods): the sizes of the subsets, imbalance ratios, etc.

Thanks for the suggestion. For TinyImageNet-LT, which has an original imbalance ratio of 100, RSMerge averages the weights of 14 models: 7 models are trained on subsets with progressively increasing imbalance ratios (from 1 to 64, doubling at each step), with 2 models per subset obtained via resampling at the same ratio. To demonstrate the impact of our proposed weight averaging scheme in RSMerge, we compare this with a WA-{imbalance ratio} baseline that averages 14 models, each trained with a fixed imbalance. Notably, WA-100 aligns with the popular model soup approach (Wortsman et al., 2022a), where the weights of 14 fully fine-tuned models on the entire dataset are averaged.

As noted by the reviewer, weight averaging consistently outperforms full fine-tuning, regardless of the subset design. However, results show that different imbalance ratios yield varying outcomes across head and tail categories. For example, WA-{8} achieves the highest tail accuracy of 75.0, whereas WA-{Full} reaches the highest head accuracy of 85.9. Rather than optimizing for a single imbalance ratio, RSMerge applies weight averaging across the full spectrum, effectively merging the advantages of both approaches to achieve a more balanced overall trade-off.

---

Q2. Effect of LA loss.

The table below demonstrates the impact of replacing LA loss with either a cross-entropy (CE) or a class-balanced sampling (CB) approach. Unlike LIFT, which relies heavily on LA loss for optimal performance, RSMerge is only partially sensitive to the choice of loss function—thanks to its inherent balance achieved through subsampling and weight averaging.

---

Q3. An analysis of the contribution of the two stages separately. This would help the reader see the contribution of the different elements of the approach.

Figure 4 of the paper breaks down the contribution of each stage in the RSMerge pipeline. To further clarify these contributions, we present the contents of the Figure in a simplified tabular format below. In particular, the introduction of EMA enhances performance for both head and tail classes while slightly reducing weight magnitude. Progressive subsampling, which averages multiple models trained on less imbalanced distributions, effectively limits weight changes and boosts tail-class accuracy—albeit at the cost of reduced head-class performance. Resampling and classifier re-training then restore head-class accuracy by averaging models trained on a fixed imbalance ratio and fine-tuning the classifier on the full dataset, respectively. Notably, RSMerge achieves a slightly higher weight magnitude than LIFT, which we argue contributes to its superior final performance by striking an optimal balance between adaptation and stability.

---

Q4. It would be valuable to include the performance of the models from the initial stage when trained on the different subsets with different imbalance factors.

Please refer to the answer to Q1.

---

Q5. How does RSMerge "bridge head and tail classes"?

Due to space constraints, we refer you to rev. EBKi rebuttal response for “Q2. Why the proposed method can solve this problem”.

---

Q6. Other Comments Or Suggestions.

Per the reviewers’ request, we have implemented the suggested changes, including centering the titles in Figure 1, correcting the typo in L294, updating the caption in Figure 2, and highlighting the two stages of RSMerge in Figure 2.

We thank the reviewer for their feedback and address their concerns in detail below:

---

Q1. Please cite proper reference for the statement that “LoRA enhances tail-class performance by maintaining weight close to the pre-trained initialization, yet it sacrifices head-class accuracy in Section 4.

This is one of our key findings, one that has not been observed in previous LT literature. As demonstrated and empirically verified in Sections 3.2 and 4.3 (weight magnitude analysis) of the paper, this phenomenon is novel within the long-tail context. Notably, at the end of Section 3.2 (L204), we also observe that our findings align with recent trends in the LLM literature, where low-rank approximation methods such as LoRA often underperform full fine-tuning while better preserving the base model's performance on tasks outside the target domain (Biderman et al., 2024).

---

Q2. There are various hyperparameters such as \lambda, M, N, and \rho. The authors need to show how the model is sensitive to those parameters.

First, note that $\rho$ is not a hyperparameter; it is a dataset-dependent variable that quantifies the proportion of head-to-tail classes (see Def. 3.1). Additionally, Section 5.2 of the paper outlines the rationale behind our choice of hyperparameters. To further support our design decisions, we conduct an ablation study on TinyImageNet-LT, which has an original imbalance ratio of 100, examining:

• $N$ represents the number of subsampling steps, ranging from 1 corresponding to the perfectly balanced dataset, up to $\log$(imbalance ratio). By convention, we double the imbalance ratio at each step, though alternative curriculums are possible.
• $M$ denotes the number of models per subset obtained via resampling at the same ratio.
• $\lambda$: the factor controlling the preservation of previously merged knowledge through progressive subsampling.

The table below summarizes our findings. As the subset imbalance ratio increases, head-class performance consistently improves improves—from 74.5 at $N=1$ to 84.8 at $N=8$. In contrast, the highest tail-class performance of 76.1 is achived at $N=4$. The best overall trade-off between head and tail performance occurs at $N=7$, indicating a balanced configuration. A similar trend is observed for weighting paramter $\lambda$: a higher value emphasizes earlier subsets with lower imbalance ratios, benefiting tail classes, while lower value shifts focus toward head-class performance. Lastly, resampling helps recover information lost due to subsampling, as reflected in the performance gain from 78.1 to 78.3.

---

Q3. Many papers related to long-tailed recognition are ignored, especially the loss function perspective (e.g., Equalization Loss v2, Balanced Softmax).

As noted in L107, the claim made by the reviewer that we ignored “Balanced Softmax” is incorrect: it is well known in the long-tail community that Balanced Softmax is a special case of the logit adjustment loss function, with the temperature fixed at 1.

---

Q.4 I recommend that authors write a section for general long-tailed recognition not limited to architectural approaches.

There is an extensive body of literature on long-tail and imbalanced recognition approaches. However, due to space constraints, we focus on the methods most closely related to our work in the main text; in response to the reviewer's request, we have expanded the literature review in the appendix.

---

Q5. Other Comments Or Suggestions.

We have corrected the citation formatting errors and typos.

Although each model learns a portion of the entire dataset, the authors need to train MN models. This might require a much longer time of training compared to other methods. Please measure the training time compared to other methods. Considering this overhead, the performance improvement is marginal or even slightly worse than one of the baseline methods (LiFT). 


How to determine $\lambda$ in Eq. 5

We appreciate the reviewer's careful reading, positive feedback on our method's accessibility and effectiveness, and constructive questions that spurred additional experiments. Below, we address their questions in detail:

---

Q1. Lack of a comprehensive description of the proposed method in the abstract.

We will follow your suggestion and improve the description of our method in the abstract.

---

Q2. Why the proposed method can solve this problem.

Motivation: Section 3.2 reveals that head and tail performances are inherently opposed, driven not only by the imbalance ratio but also by our introduced head-tail ratio. Most LT methods boost tail accuracy at the cost of head-class performance. For instance, on CIFAR100-LT with a head-tail ratio of 19 (95 head classes) (Fig. 2), LIFT improves tail results via low-rank adaptation but underfits head classes, resulting in an overall performance of 84.9 versus 86.3 for Full-FT. This trade-off suggests that while full-rank adaptation excels at capturing head-class information, it often sacrifices tail performance by straying from the pre-trained configuration.

Our Objective: We aim to sustain robust performance across both head and tail classes, irrespective of the head-tail scenario—a challenge that has been largely unexplored in LT literature. Our goal is twofold: we want the flexibility of full-rank optimization to effectively capture head-class information, and we encourage balanced learning between head and tail classes to prevent the model from being biased toward head classes.

Philosophy of RSMerge: RSMerge merges full-rank optimization with balanced learning via progressive subsampling. While subsampling improves tail-class performance (Chaudhuri et al., 2023), it loses head-class data, harming head performance. We address this by training independent models on subsets with increasing imbalance ratios. Each model specializes in a segment of the imbalance spectrum, and their average produces a final model robust across the range. We substantiate this insight with a detailed ablation study below.

Empirical evidence: RSMerge averages the weights of $N \times M$ models, where $N$ is the number of subsampling steps (from 1 for a perfectly balanced dataset up to $\log$(imbalance ratio), doubling the ratio at each step) and $M$ is the number of models per subset via resampling. For TinyImageNet-LT (imbalance ratio 100), we use $N=7$ and $M=2$. The table below compares RSMerge with WA-{imbalance ratio} baselines—each averaging 14 models trained with a fixed imbalance ratio. Notably, WA-100 corresponds to the popular model soup approach (Wortsman et al., 2022a).

Results show that different imbalance ratios yield varying outcomes across head and tail categories. For example, WA-{8} achieves the highest tail accuracy of 75.0, whereas WA-{100} reaches the highest head accuracy of 85.9. Rather than optimizing for a single imbalance ratio, RSMerge applies weight averaging across the full spectrum, effectively merging the advantages of both approaches to achieve a more balanced overall trade-off.

---

Q3. Refinement of Figure 3.

Figure 3 illustrates the overall training steps involved in RSMerge. To enhance clarity, we explicitly highlight Stages 1 and 2 of the pipeline in the figure.

---

Q4. Computational analysis.

For space reasons, we refer you to rev. 5wyk rebuttal response for “Q2. Computational analysis”.

---

Q5. Lack of algorithm’s implementation details.

We describe our algorithm in Section 4, and detail hyperparameters, experimental setup, and additional information in Section 5.2 and Appendices A and B. The source code is also available. Please let us know which implementation details are unclear or hinder reproducibility.

---

Q6. The performance improvement of the proposed method is relatively limited.

Addressing class imbalance requires considering the head-to-tail ratio—a critical but often overlooked factor. While traditional benchmarks focus on tail-dominated scenarios, real-world datasets like CXR-LT have a majority of head classes. In such cases, enhancing tail performance without sacrificing head accuracy is essential. Our method, RSMerge, achieves this by combining full-parameter fine-tuning for head-class information with weight averaging across progressively subsampled subsets, resulting in a better trade-off (RSMerge = 39.3 vs. LIFT = 38.5).